Reproduce “Object Hallucination in Image Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, Kate Saenko
Researchers at UC Berkeley and Boston University introduced CHAIR (Caption Hallucination Assessment with Image Relevance), a new metric to quantify object hallucination in image captions. Their analysis revealed that 7.4% to 17.7% of generated sentences contain hallucinated objects, finding that attention models generally reduce hallucination, while self-critical training can increase it despite higher standard metric scores.
Key insights
- Stronger visual grounding, particularly through access to spatial features and explicit object proposals, is more crucial for reducing hallucination than attention mechanisms alone.
- Hallucinations are primarily driven by an over-reliance on language priors rather than true visual misclassification, especially in models with higher hallucination rates.
- Optimizing image captioning models solely for standard metrics like CIDEr can paradoxically increase object hallucination.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.