ExAct: A Video-Language Benchmark for Expert Action Analysis

Han Yi (University of North Carolina at Chapel Hill) · Yulu Pan (University of North Carolina at Chapel Hill) · Feihong He (University of North Carolina at Chapel Hill) · Xinyu Liu (University of Virginia) · Benjamin Zhang (University of North Carolina at Chapel Hill) · Oluwatumininu Oguntola (University of North Carolina at Chapel Hill) · Gedas Bertasius (UNC Chapel Hill)

accuracy comparisondataset availabilitydeveloping vlmsexpert-curated video question-answer pairsexpert-level understandingfine-grained evaluationgemini 2.5 pro modelnuanced understandingperformance gapphysical and procedural domainsphysical human activitiesprecise understandingstate-of-the-art vlmstrained human expertsvideo-language benchmark

We present ExAct, a new video-language benchmark for expert-level understanding of skilled physical human activities. Our new benchmark contains 3,521 expert-curated video question-answer pairs spanning 11 physical activities in 6 domains: Sports, Bike Repair, Cooking, Health, Music, and Dance. ExAct requires the correct answer to be selected from five carefully designed candidate options, thus necessitating a nuanced, fine-grained, expert-level understanding of physical human skills. Evaluating the recent state-of-the-art VLMs on ExAct reveals a substantial performance gap relative to human expert performance. Specifically, the best-performing Gemini 2.5 Pro model achieves only 55.35% accuracy, well below the 82.02% attained by trained human experts. We believe that ExAct will be beneficial for developing and evaluating VLMs capable of precise understanding of human skills in various physical and procedural domains. Dataset and code are available at https://texaser.github.io/exact_project_page/.