OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model

Jingya Wang (ShanghaiTech University) · Qi Ye (Zhejiang University) · Zhenhao Zhang (ShanghaiTech University) · Ye Shi (ShanghaiTech University) · Lingxiao Yang (ShanghaiTech University) · Suting Ni (ShanghaiTech University)
3d hand-object interactionsaffordance alignment optimizationaffordance-driven diffusion modelcomplex instructions breakdowncomplex language instructionsfree-form language commandsgeneralization to novel object categoriesjoint affordance groundinglocalization of interaction regionslong-horizon manipulation sequencesmulti-stage tasksmultimodal large language modelopen-world hoi synthesispenetration minimizationsemantic task decompositiontraining-free physics refinement

Understanding and synthesizing realistic 3D hand-object interactions (HOI) is critical for applications ranging from immersive AR/VR to dexterous robotics. Existing methods struggle with generalization, performing well on closed-set objects and predefined tasks but failing to handle unseen objects or open-vocabulary instructions. We introduce OpenHOI, the first framework for open-world HOI synthesis, capable of generating long-horizon manipulation sequences for novel objects guided by free-form language commands. Our approach integrates a 3D Multimodal Large Language Model (MLLM) fine-tuned for joint affordance grounding and semantic task decomposition, enabling precise localization of interaction regions (e.g., handles, buttons) and breakdown of complex instructions (e.g., “Find a water bottle and take a sip”) into executable sub-tasks. To synthesize physically plausible interactions, we propose an affordance-driven diffusion model paired with a training-free physics refinement stage that minimizes penetration and optimizes affordance alignment. Evaluations across diverse scenarios demonstrate OpenHOI’s superiority over state-of-the-art methods in generalizing to novel object categories, multi-stage tasks, and complex language instructions.