Reproduce “Mitigating Open-Vocabulary Caption Hallucinations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, Hadar Averbuch-Elor
Researchers from Tel-Aviv University developed OpenCHAIR, the first open-vocabulary benchmark for evaluating object hallucinations in image captioning, and MOCHa, a reinforcement learning framework. MOCHa reduces open-vocabulary hallucination rates in state-of-the-art captioning models, such as decreasing BLIP-L's OpenCHAIR score from 19.2 to 18.3, while preserving or enhancing standard caption quality metrics.
Key insights
- Open-vocabulary hallucination is a more realistic and challenging problem than closed-vocabulary, requiring new evaluation methods beyond fixed object lists.
- Reinforcement Learning, particularly with a carefully designed multi-objective reward function, can effectively optimize for complex sequence-level properties like factual groundedness.
- Leveraging powerful foundation models (LLMs, text-to-image diffusion models) can automate the creation of diverse, high-quality synthetic data for benchmarking open-vocabulary challenges, reducing reliance on manual annotation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.