InstructHOI: Context-Aware Instruction for Multi-Modal Reasoning in Human-Object Interaction Detection

Zhiyong Wang (Harbin Institute of Technology, Shenzhen) · Jinguo Luo (Harbin Institute of Technology, Shenzhen) · Weihong Ren (Harbin Institute of Technology, Shenzhen) · Quanlong Zheng (Guangdong OPPO Mobile Telecommunications Corp.,Ltd.) · Yanhao Zhang (Oppo AI center) · Zhenlong Yuan (, Chinese Academy of Sciences) · Haonan Lu (OPPO Guangdong Mobile Telecommunications Co., Ltd.) · Honghai LIU (Harbin Institute of Technology, Shenzhen)
adaptive filteringcontext-aware instruction generatorcontext-aware instructionshoi-domain fine-tuninghuman-object interactioninstructhoiinteraction-reasoning image-text pairsinterest token selectorlarge foundation modelslinguistic instructionsmulti-modal reasoningprompt-based approachesreasoning capabilitiesstate-of-the-art performancevisual interactive contextzero-shot settings

Recently, Large Foundation Models (LFMs), e.g., CLIP and GPT, have significantly advanced the Human-Object Interaction (HOI) detection, due to their superior generalization and transferability. Prior HOI detectors typically employ single- or multi-modal prompts to generate discriminative representations for HOIs from pretrained LFMs. However, such prompt-based approaches focus on transferring HOI-specific knowledge, but unexplore the potential reasoning capabilities of LFMs, which can provide informative context for ambiguous and open-world interaction recognition. In this paper, we propose InstructHOI, a novel method that leverages context-aware instructions to guide multi-modal reasoning for HOI detection. Specifically, to bridge knowledge gap and enhance reasoning abilities, we first perform HOI-domain fine-tuning on a pretrained multi-modal LFM, using a generated dataset with 140K interaction-reasoning image-text pairs. Then, we develop a Context-aware Instruction Generator (CIG) to guide interaction reasoning. Unlike traditional language-only instructions, CIG first mines visual interactive context at the human-object level, which is then fused with linguistic instructions, forming multi-modal reasoning guidance. Furthermore, an Interest Token Selector (ITS) is adopted to adaptively filter image tokens based on context-aware instructions, thereby aligning reasoning process with interaction regions. Extensive experiments on two public benchmarks demonstrate that our proposed method outperforms the state-of-the-art ones, under both supervised and zero-shot settings.