Universal Visuo-Tactile Video Understanding for Embodied Interaction

Fei Yu (The Chinese University of Hong Kong, Shenzhen) · Fei Ma (Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), also known as Guangming Laboratory) · Yifan Xie (Tsinghua University) · Mingyang Li (Tsinghua University) · Shoujie Li (Tsinghua University, Tsinghua University) · Xingting Li (University of Science and Technology Beijing) · Guangyu Chen (SUN YAT-SEN UNIVERSITY) · Wenbo Ding (Tsinghua University, Tsinghua University)
comparative analysiscross-modal integrationcross-sensor integrationembodied agentshaptic feedbackhuman-machine interactionmulti-modal large language modelnatural language generationtactile attributestactile perceptiontactile reasoning capabilitiesvisuo-tactile representationvisuo-tactile videovtv-text alignmentvtv150k dataset

Tactile perception is essential for embodied agents to understand the physical attributes of objects that cannot be determined through visual inspection alone. While existing methods have made progress in visual and language modalities for physical understanding, they fail to effectively incorporate tactile information that provides crucial haptic feedback for real-world interaction. In this paper, we present VTV-LLM, the first multi-modal large language model that enables universal Visuo-Tactile Video (VTV) understanding, bridging the gap between tactile perception and natural language. To address the challenges of cross-sensor and cross-modal integration, we contribute VTV150K, a comprehensive dataset comprising 150,000 video frames from 100 diverse objects captured across three different tactile sensors (GelSight Mini, DIGIT, and Tac3D), annotated with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction). We develop a novel three-stage training paradigm that includes VTV enhancement for robust visuo-tactile representation, VTV-text alignment for cross-modal correspondence, and text prompt finetuning for natural language generation. Our framework enables sophisticated tactile reasoning capabilities including feature assessment, comparative analysis, and scenario-based decision-making. Extensive experimental evaluations demonstrate that VTV-LLM achieves superior performance in tactile reasoning tasks, establishing a foundation for more intuitive human-machine interaction in tactile domains.