Reproduce “ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianjiang Yang, Yanshu li, Ziyan Huang
ReLoop introduces a closed-loop training framework for Multimodal Large Language Models (MLLMs) that incorporates internal consistency feedback to reduce hallucinations. The system improves factual consistency and visual grounding by simulating human-like self-reflection, leading to substantial reductions in hallucination rates and enhanced cross-modal faithfulness across diverse MLLM architectures.
Key insights
- Hallucinations in MLLMs stem from structured, modality-specific representation gaps, with different hallucination types correlating to specific internal consistency signal deficiencies.
- Integrating internal, feedback-driven supervision directly into the MLLM training process, inspired by human self-reflection, is effective for addressing the root causes of hallucinations.
- Combining multi-level supervision signals (linguistic, visual, and attention consistency) significantly enhances multimodal grounding and factual consistency, making the model more robust to noisy training inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.