Reproduce “Debiasing Multimodal Large Language Models via Penalization of Language Priors” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

YiFan Zhang, Yang Shi, Weichen Yu, Qingsong Wen, Xue Wang, Wenjing Yang, Zhang Zhang, Liang Wang and 1 more

Researchers developed training-free strategies, Post-Hoc Debias and Visual Debias Decoding, to mitigate hallucination in Multimodal Large Language Models (MLLMs) by penalizing their bias towards language priors. The work also demonstrated that systematic optimization of decoding configurations can improve MLLM performance by up to 20 percentage points and influence benchmark fairness.

Key insights

  • MLLMs exhibit a profound and consistent bias towards language priors, even generating confident answers with absent or noisy visual inputs, driven by imbalanced attention to text over visual tokens.
  • Training-free debiasing strategies effectively mitigate MLLM hallucinations and improve visual grounding, offering practical and efficient solutions for enhancing model reliability.
  • MLLM performance is highly sensitive to generative decoding configurations, where systematic optimization can yield considerable improvements and reveal limitations in current evaluation benchmarks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.