Reproduce “Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution” on OpenResearch

About this paper

Weiming Ren, Raghav Goyal, Zhiming Hu, Tristan Ty Aumentado-Armstrong, Iqbal Mohomed, Alex Levinshtein

Researchers from the University of Waterloo and Samsung Electronics developed a "Hallucination Score" using Multimodal Large Language Models to quantify perceptual inaccuracies in Generative Super-Resolution (GSR) outputs. They then used efficient differentiable proxies of this score as a reward function to fine-tune diffusion-based GSR models, substantially increasing their hallucination scores (e.g., GPT-HS improved from 2.99 to 3.87 for SeeSR) while preserving perceptual quality.

Key insights

• Multimodal Large Language Models (MLLMs) can effectively characterize complex perceptual artifacts like hallucinations in generative image models, providing scores that strongly correlate with human judgment.
• Existing fidelity and perceptual IQA metrics are inadequate for assessing hallucinations in GSR, often showing poor or negative correlation with human perception of these specific artifacts.
• Differentiable proxies trained on MLLM-derived scores enable scalable and targeted fine-tuning of generative models, allowing for the reduction of specific undesirable artifacts without compromising overall perceptual quality.

Summary by alphaXiv.