Reproduce “Re-Thinking the Automatic Evaluation of Image-Text Alignment in Text-to-Image Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huixuan Zhang, Xiaojun Wan
Researchers from Peking University's Wangxuan Institute of Computer Technology demonstrated that widely used text-to-image evaluation metrics, including CLIPScore, VQAScore, and DSGScore, lack robustness to minor image perturbations and randomness inherent in generation, and often report statistically significant yet practically negligible differences between models. Their empirical analysis revealed that these metrics frequently yield inconsistent model rankings and show extreme sensitivity to imperceptible image changes, undermining their reliability for benchmarking.
Key insights
- Text-to-image evaluation metrics require validation beyond human correlation, needing assessment for robustness and significance to be trustworthy.
- Small numerical differences in current automatic metrics often stem from measurement instability rather than true performance gains.
- Relying solely on existing metrics for model optimization may lead to misdirected research and wasted resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.