Reproduce “Evaluating Image Caption via Cycle-consistent Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyu Cui, Jinbin Bai, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Ye Shi
CAMScore, a novel evaluation metric for image captioning, utilizes cycle-consistent text-to-image generation to compare generated captions against original images within the visual modality. This approach consistently achieved superior correlation with human judgments and state-of-the-art pairwise ranking accuracy across various benchmarks.
Key insights
- Leveraging cycle consistency through text-to-image generation allows for robust image caption evaluation within a single modality, effectively mitigating the 'modality gap' of direct cross-modal comparisons.
- A multi-level evaluation framework, particularly incorporating semantic and object-level analysis (including spatial and depth information), is crucial for achieving high correlation with nuanced human judgments.
- The pixel-level component contributes less to human judgment correlation compared to semantic and object-level aspects, indicating humans prioritize content and relationships over exact visual fidelity when assessing captions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.