Reproduce “Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, William Yang Wang
Researchers at the University of California, Santa Barbara introduced T2IScoreScore (TS2), an objective benchmark and meta-metrics framework for evaluating text-to-image prompt coherence metrics. Their analysis revealed that simpler embedding-correlation metrics, like CLIPScore, performed comparably to or even better than more complex, computationally intensive VLM-based methods on objective semantic error detection, often being Pareto-optimal for performance and cost.
Key insights
- Simpler embedding-correlation metrics (CLIPScore, ALIGNScore) performed comparably to, or even outperformed, more complex and computationally expensive VLM-based metrics on objective semantic error detection.
- Many advanced, high-cost T2I faithfulness metrics do not offer a substantial, objective performance advantage over more efficient methods when assessed on their ability to identify and rank semantic errors.
- The 'Real' and 'Nat' subsets of the benchmark, representing authentic T2I model failures and natural images, proved significantly more challenging for metrics than synthetically designed errors, indicating limitations in handling real-world complexity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.