Reproduce “MLLMs are Deeply Affected by Modality Bias” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Zheng, Chenfei Liao, Yuqian Fu, Kaiyu Lei, Yuanhuiyi Lyu, Lutao Jiang, Bin Ren, Jialei Chen and 10 more
This paper provides a comprehensive analysis of modality bias in Multimodal Large Language Models (MLLMs), where models disproportionately rely on textual information over visual inputs. The research identifies key contributing factors such as dataset imbalances and architectural choices, offering a systematic experimental validation and a roadmap for addressing this fundamental challenge.
Key insights
- Modality bias in Multimodal Large Language Models (MLLMs) is defined as an asymmetric reliance on different input modalities, often favoring text over visual or other inputs.
- Three primary factors contribute to this bias: dataset imbalances (text abundance), asymmetric modal backbone capabilities (pre-trained language models), and training objectives (prioritizing text generation).
- Secondary factors include different convergence rates for modalities and the specific methods used for modal interaction and integration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.