MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence

Wentong Li (Zhejiang University) · Limin Wang (Nanjing University) · Jinwei Hu (Nanjing University of Aeronautics and Astronautics) · Shiping Ge (Nanjing University) · Jie Qin (Inception Institute of Artificial Intelligence) · Yue Feng (Nanjing University of Aeronautics and Astronautics) · Qijia Lu (Nanjing University of Aeronautics and Astronautics) · Jiawei Niu (Nanjing University of Aeronautics and Astronautics) · Li Tan (Nanjing University of Aeronautics and Astronautics) · Shuo Yuan (Nanjing University of Aeronautics and Astronautics) · Ziyi Yan (Nanjing University of Aeronautics and Astronautics) · Yizhen Jia (Nanjing University of Aeronautics and Astronautics) · Qingzhi He (Nanjing University of Aeronautics and Astronautics) · Ethan Chen (Hong Kong Polytechnic University)
comprehensive evaluationfine-grained retrievalimage-based vlmslong-video platformsmllmsmulti-level visual correspondencemulti-modal untrimmed video retrievalquestion-answering formatreranking scoreretrieval matching criteriastate-of-the-art methodsuntrimmed videosvideo retrieval modelsvideo-centric queries

We propose the Multi-modal Untrimmed Video Retrieval task, along with a new benchmark (MUVR) to advance video retrieval for long-video platforms. MUVR aims to retrieve untrimmed videos containing relevant segments using multi-modal queries. It has the following features: **1) Practical retrieval paradigm:** MUVR supports video-centric multi-modal queries, expressing fine-grained retrieval needs through long text descriptions, video tag prompts, and mask prompts. It adopts a one-to-many retrieval paradigm and focuses on untrimmed videos, tailored for long-video platform applications. **2) Multi-level visual correspondence:** To cover common video categories (e.g., news, travel, dance) and precisely define retrieval matching criteria, we construct multi-level visual correspondence based on core video content (e.g., news events, travel locations, dance moves) which users are interested in and want to retrieve. It covers six levels: copy, event, scene, instance, action, and others. **3) Comprehensive evaluation criteria:** We develop 3 versions of MUVR (i.e., Base, Filter, QA). MUVR-Base/Filter evaluates retrieval models, while MUVR-QA assesses MLLMs in a question-answering format. We also propose a Reranking Score to evaluate the reranking ability of MLLMs. MUVR consists of 53K untrimmed videos from the video platform Bilibili, with 1,050 multi-modal queries and 84K matches. Extensive evaluations of 3 state-of-the-art video retrieval models, 6 image-based VLMs, and 10 MLLMs are conducted. MUVR reveals the limitations of retrieval methods in processing untrimmed videos and multi-modal queries, as well as MLLMs in multi-video understanding and reranking. Our code and benchmark is available at https://github.com/debby-0527/MUVR.