Reproduce “Understanding Multimodal Hallucination with Parameter-Free Representation Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yueqian Wang, Jianxin Liang, Yuxuan Wang, Huishuai Zhang, Dongyan Zhao
Peking University researchers introduced Pfram, a parameter-free metric, to investigate object hallucination in MLLMs by quantitatively comparing internal image representations with human annotations. Their work reveals a strong correlation between the quality of object information in these representations and the model's object hallucination rate, establishing a diagnostic tool to pinpoint architectural origins of such errors.
Key insights
- The quality of object information preserved within MLLM image representations, quantified by Pfram(F; G_obj), strongly and consistently correlates with the model's object hallucination performance.
- Factors like general image understanding (Pfram(F; G_des)), LLM size, or modality projector type did not show a strong correlation with object hallucination, suggesting specific object grounding is paramount.
- Pfram can diagnose information flow; visual encoders gradually improve object information, while LLM components primarily process existing features, and inter-module information loss can contribute to hallucination.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.