Reproduce “Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Meiqi Chen, Yixin Cao, Yan Zhang, Chaochao Lu
Researchers from Peking University, Fudan University, and Shanghai AI Lab developed a causal framework to quantify unimodal biases in Multimodal Large Language Models (MLLMs), introduced the challenging MORE dataset, and proposed CAVE, a causality-enhanced agent framework. This work demonstrated that leading MLLMs struggle significantly with multi-hop reasoning on MORE, and CAVE improved their accuracy by up to 10.9 percentage points.
Key insights
- MLLMs exhibit high sensitivity to changes in ground truth but low robustness, with predictions fluctuating significantly even with irrelevant changes to textual or visual cues.
- Over 40% of incorrect answers by MLLMs on the MORE dataset were directly attributable to either language or vision bias options, demonstrating a pervasive over-reliance on unimodal shortcuts.
- MLLMs can show 'homologous biases,' where their language component favors distractors generated by similar underlying language models, indicating model-specific bias patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.