Reproduce “See or Guess: Counterfactually Regularized Image Captioning” on OpenResearch

About this paper

Qian Cao, Xu Chen, Ruihua Song, Xiting Wang, Xinting Huang, Yuchen Ren

Researchers from Renmin University of China and Tencent AI Lab introduce a counterfactually regularized framework for image captioning that significantly reduces object hallucination and enhances interpretability by leveraging causal inference. This approach improves robustness across various image captioning models and datasets without sacrificing performance on factual images.

Key insights

• Applying counterfactual causal inference can effectively reduce object hallucination in image captioning models by forcing them to rely more on direct visual evidence.
• The Natural Direct Effect (NDE) regularization specifically enhances model interpretability by establishing clearer, more precise visual-textual alignments, showing which image regions contribute to specific word generation.
• Integrating causal regularization into existing models improves their robustness to biases and counterfactual interventions while maintaining or even improving performance on factual images.

Summary by alphaXiv.