Reproduce “BIMA: Bijective Maximum Likelihood Learning Approach to Hallucination Prediction and Mitigation in Large Vision-Language Models” on OpenResearch

About this paper

Huu-Thien Tran, Thanh-Dat Truong, Khoa Luu

The BIMA (Bijective Maximum Likelihood Learning) approach utilizes normalizing flow theories to establish a bijective mapping between generated responses and a learned reference distribution of non-hallucinated content, integrating this mechanism directly into the fine-tuning of Large Vision-Language Models. This method reduces hallucination, achieving a 7.6% improvement in CHAIR_S score compared to prior best methods and demonstrating competitive performance on the POPE benchmark.

Key insights

• Ground-truth responses from training data can serve as a robust 'reference distribution' for non-hallucinated content, providing a target for model outputs.
• Normalizing flow theories can establish a bijective mapping between model-generated responses and this reference distribution, enabling a quantifiable measure of how 'hallucinated' a response is.
• Integrating a bijective metric during instruction fine-tuning allows LVLMs to inherently learn to produce responses that align with the non-hallucinated reference distribution, addressing hallucination at a foundational training level rather than solely at inference.

Summary by alphaXiv.