HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios

Jiamin Wu (The Chinese University of Hong Kong) · Junwei Zheng (Karlsruhe Institute of Technology) · Yufan Chen (Karlsruhe Institute for Technology) · Kunyu Peng (Karlsruhe Institute for Technology) · Kailun Yang (Hunan University) · Rainer Stiefelhagen (Karlsruher Institut für Technologie) · Junchao Huang (Beijing Institute of Technology) · Xiangsheng Huang (Institute of Automation, Chinese Academy of Sciences) · Di Wen (Karlsruher Institut für Technologie) · Chongqing Hao (Hebei University of Science and Technology)
action segmentationaction segmentation generationcross-input gate attentional xlstmevaluation settingsfine-grained actionsfourier-conditioned diffusion frameworkholistic-partial awarelong-range reasoningmulti-person scenariosreferring human action segmentationrhas133state-of-the-art resultstextual reference-guidedvisual cues aggregationvlm-based feature extractors

Action segmentation is a core challenge in high-level video understanding, aiming to partition untrimmed videos into segments and assign each a label from a predefined action set. Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual reference-guided human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Benchmarking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose a holistic-partial aware Fourier-conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-art results on RHAS133 in diverse evaluation settings. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF.