multimodal models
AI models designed to process and understand data from multiple modalities (e.g., text, image, audio) simultaneously. They are capable of capturing the interrelations among different types of data for improved understanding and generation tasks.
- Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
- Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
- Data Selection Matters: Towards Robust Instruction Tuning of Large Multimodal Models
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
- Dual-Space Semantic Synergy Distillation for Continual Learning of Unlabeled Streams
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
- MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
- OpenMMEgo: Enhancing Egocentric Understanding for LMMs with Open Weights and Data
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
- Quantifying Cross-Modality Memorization in Vision-Language Models
- RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
- RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You