Reproduce “Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Mi, Licheng Pan, Jiayi Zhu, Qirui Liu, Yitong Qiao, Changtong Xu, Tao Wei, Fei Sun and 2 more
CaRe addresses audio-visual errors where a correct modality-specific answer is lost in the joint prediction. A text-only probe nominates audio or video; if the unimodal and joint answers disagree, CaRe steers the joint representation toward the unimodal reference while retaining both modalities. On CMM, MiniCPM-o-2.6 reaches 86.9% accuracy versus MAD’s 77.3%; evidence is limited to discrete-answer cross-modal hallucination benchmarks.
Key insights
- A substantial share of joint-branch errors are recoverable from unimodal branches: reported recovery rates reach 74.4% on CMM and 46.2% on AVHBench.
- Steering the joint representation toward unimodal evidence can outperform simply replacing the joint answer with the unimodal answer, by as much as 10.82 percentage points in reported comparisons.
- The method requires extra inference passes and is designed for settings with finite answer choices and identifiable modality conflicts; it does not address every hallucination type.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.