Reproduce “Energy-Guided Decoding for Object Hallucination Mitigation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xixi Liu, Ailin Deng, Christopher Zach
Researchers at Chalmers University of Technology and National University of Singapore introduced an energy-guided decoding method to reduce object hallucinations in large vision-language models. The approach dynamically selects the most reliable internal decoder layer based on energy scores, leading to improved factual consistency and up to 10.2% higher accuracy on POPE benchmarks while mitigating "Yes" ratio bias.
Key insights
- VLMs possess an inherent "Yes" ratio bias, driven by language priors, leading to over-affirmative responses, particularly in challenging VQA scenarios.
- Standard decoding exhibits an overconfidence in "Yes" answers compared to "No" answers, indicating a lack of calibration in VLM predictions.
- Utilizing an energy score allows for the dynamic selection of an internal decoder layer, often the penultimate layer, which provides a more reliable and calibrated logit distribution for token generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.