Reproduce “Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations” on OpenResearch

About this paper

Yifan Lu, Ziqi Zhang, Chunfeng Yuan, Jun Gao, Congxuan Zhang, Xiaojuan Qi, Bing Li, Weiming Hu

Researchers from CASIA developed Autonomous Preference Alignment via Self-Injection (APASI), a self-sufficient framework for Large Vision-Language Models (LVLMs) that autonomously generates preference data by deliberately injecting specific hallucination patterns. This approach effectively reduces hallucinations and concurrently enhances overall model capabilities without relying on external human annotations or auxiliary AI models.

Key insights

• Injecting hallucinations that realistically mimic how LVLMs naturally err (e.g., hallucinating co-occurring objects or over-relying on linguistic priors) generates highly effective and targeted learning signals for preference alignment.
• An autonomous, self-sufficient data generation mechanism, combined with iterative training and a curriculum for hallucination intensity, enables scalable and sustainable model improvement without external dependencies.
• Mitigating hallucinations through precise dis-preferred response generation not only reduces factual inconsistencies but also enhances the LVLM's general comprehensive abilities across diverse benchmarks.

Summary by alphaXiv.