Prompt-guided Disentangled Representation for Action Recognition

Liang Zhang (Renmin University of China) · tianci wu (xidian university) · Guangming Zhu (Xidian University) · Lu jiang (Xi'an University of Electronic Science and Technology) · Siyuan Wang (Xidian University) · Ning Wang (Xi'an University of Electronic Science and Technology) · Nuoye Xiong (Xi'an University of Electronic Science and Technology)

action recognitionaction-specific representationscharadescomplex video action datasetsdisentangling actionsdynamic prompt moduledynamic weightsgraph parsing neural networkmulti-action scenariosprompt-guided disentangled representationspatio-temporal scene graphssportshhistate-of-the-art methods video understandingvideo-adapted gpnn

Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Extensive experiments on two complex video action datasets, Charades and SportsHHI, demonstrate the effectiveness of our approach against state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git.