Reproduce “Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset and 1 more
The Davidsonian Scene Graph (DSG) framework enhances the reliability and interpretability of fine-grained evaluation for text-to-image generation models by producing atomic, unique, and dependency-aware questions. This approach yields a significantly higher correlation with human judgment and offers detailed diagnostics for pinpointing T2I model strengths and weaknesses.
Key insights
- Decomposing text prompts into atomic, dependency-aware propositions significantly improves the reliability and interpretability of QG/A evaluation frameworks by ensuring questions are precise and verifiable.
- While improving overall T2I evaluation metrics, incorporating question dependencies substantially enhances per-question VQA-human matching accuracy, confirming human evaluators implicitly consider these relationships.
- Current state-of-the-art VQA models still struggle with deep semantics, precise counting, accurate text rendering, and subjective or domain-specific attributes, highlighting clear areas for future VQA model development.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.