OmniBench: Towards The Future of Universal Omni-Language Models

Ge Zhang (University of Michigan - Ann Arbor) · Yiming Liang (University of the Chinese Academy of Sciences) · Zhu (Guangdong OPPO Mobile Telecommunications Corp.,Ltd.) · Yizhi Li (The University of Manchester) · Siwei Wu (Nanjing University of Science and Technology) · Zhenzhu Yang (China University of Geoscience Beijing) · Noah Wang · Yinghao Ma (Centre for Digital Music, Queen Mary University of London) · Xingwei Qu (University of Manchester) · Ruibin Yuan (Carnegie Mellon University) · Jian Yang (nanjing university) · ZHAO-XIANG ZHANG (Chinese Academy of Sciences, China) · Jiaheng Liu (Nanjing University) · Wenhao Huang (Key Laboratory of Machine Perception) · Shihao Li (nanjing university) · Yanghai Wang (nanjing university) · Hangyu Guo (Alibaba Group) · Jinjie Shi (Queen Mary, University of London) · Xinyue Zhang (National University of Singapore) · Yidan WEN (Northwest Polytechnical University Xi'an) · Ruibo Liu (Google DeepMind) · Emmanouil Benetos (Queen Mary University of London) · Chenghua Lin (University of Manchester)
baseline modelshigh-quality human annotationsinstruction tuning datasetinstruction-followingintegrated understandingmodel evaluationmultimodal large language modelsomni-language modelsomnibenchopen-source olmsperformance enhancementreasoningrobust tri-modal integrationtraining strategiestri-modal processing

Recent advancements in multimodal large language models (MLLMs) have focused on integrating multiple modalities, yet their ability to simultaneously process and reason across different inputs remains underexplored. We introduce OmniBench, a novel benchmark designed to evaluate models’ ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define language models capable of such tri-modal processing as omni-language models (OLMs). OmniBench features high-quality human annotations that require integrated understanding across all modalities. Our evaluation reveals that: i) open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts; and ii) most baseline models perform poorly (below 50% accuracy) even with textual alternatives to image/audio inputs. To address these limitations, we develop OmniInstruct, an 96K-sample instruction tuning dataset for training OLMs. We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance. Codes and data could be found at https://m-a-p.ai/OmniBench/.