Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

Zinuo Li (University of Western Australia) · Xian Zhang (University of Western Australia) · Yongxin Guo (Chinese University of HongKong, Shenzhen) · Mohammed Bennamoun (University of Western Australia) · Farid Boussaid · Girish Dwivedi (University of Western Australia) · Luqi Gong (Zhejiang Lab) · Qiuhong Ke (Monash University)
audio-visual integrationcomprehensive video interpretationdataset curationdiverse modality combinationsfine-tuned llmsflexible input combinationsholistic video analysislong-form videosmodality dropoutmultimodal capabilitiesmultimodal video analysisquery-based connectorrobust performancetriple-modality large language modelvideo temporal understanding

Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing a scene in the video like “A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and applauding” requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggle to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense, a triple-modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense's multimodal capabilities, we introduce TriSense-2M, a high-quality dataset of over 2 million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis.