Reproduce “DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Longtian Qiu, Shan Ning, Chuyu Zhang, Jiaxuan Sun, Xuming He

DA-DPO introduces a cost-efficient, difficulty-aware preference optimization framework that significantly reduces hallucinations in Multimodal Large Language Models (MLLMs). The method improves generative hallucination rates on the AMBER benchmark from 35.7 to 28.0 and cognitive accuracy from 45.3 to 59.8, while preserving the MLLMs' general multimodal capabilities.

Key insights

  • Standard multimodal DPO exhibits an "easy sample bias," causing overfitting and preventing effective learning from more challenging, nuanced preference data.
  • Complementary signals from both contrastive and generative pre-trained Vision-Language Models (VLMs) can be effectively combined in a cost-efficient manner to implicitly estimate the inherent difficulty of preference samples.
  • Dynamically scaling the DPO loss based on estimated sample difficulty leads to more balanced learning, which effectively mitigates hallucinations while preserving general MLLM capabilities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.