VMDT: Decoding the Trustworthiness of Video Foundation Models

Bo Li (Nanjing University) · Dawn Song (UC Berkeley) · Yujin Potter (University of California, Berkeley) · Zhun Wang (University of California, Berkeley) · Nicholas Crispino (University of California, Santa Cruz) · Kyle Montgomery (University of California, Santa Cruz) · Alexander Xiong (University of California, Berkeley) · Ethan Chang (University of Illinois at Urbana-Champaign) · Francesco Pinto (University of Chicago) · Yuqi Chen (University of California, Santa Cruz) · Rahul Gupta (Amazon.com) · Morteza Ziyadi (Amazon) · Christos Christodoulopoulos (Information Commissioner's Office) · Chenguang Wang (University of California Berkeley)
adversarial robustnessfairnessfoundation modelshallucinationharmful queriesmodel scaleopen-source modelsprivacy riskssafety dimensionssystematic frameworktext-to-video modelstrustworthiness benchmarksunfairnessvideo foundation modelsvideo-modal decodingtrustvideo-to-text models

As foundation models become more sophisticated, ensuring their trustworthiness becomes increasingly critical; yet, unlike text and image, the video modality still lacks comprehensive trustworthiness benchmarks. We introduce VMDT (Video-Modal DecodingTrust), the first unified platform for evaluating text-to-video (T2V) and video-to-text (V2T) models across five key trustworthiness dimensions: safety, hallucination, fairness, privacy, and adversarial robustness. Through our extensive evaluation of 7 T2V models and 19 V2T models using VMDT, we uncover several significant insights. For instance, all open-source T2V models evaluated fail to recognize harmful queries and often generate harmful videos, while exhibiting higher levels of unfairness compared to image modality models. In V2T models, unfairness and privacy risks rise with scale, whereas hallucination and adversarial robustness improve