Reproduce “Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muxi Chen, Yi Liu, Jian Yi, Changran Xu, Qiuxia Lai, Hongliang Wang, Tsung-Yi Ho, Qiang Xu
A comprehensive, nuanced framework for evaluating text-to-image generative models in human image synthesis is introduced, assessing aesthetic quality, defect rates, concept coverage, and fairness. The evaluation reveals Midjourney's superior aesthetic and lower defect rates compared to Stable Diffusion models, while also identifying persistent issues like distorted hands and widespread biases across gender, race, and age in all examined models.
Key insights
- Midjourney consistently produces images with higher aesthetic scores and lower defect rates for human faces compared to Stable Diffusion models, indicating superior realism and visual appeal.
- While Stable Diffusion models show iterative improvements in aesthetic scores and defect rates, persistent challenges remain, particularly with generating realistic hands across all evaluated models (around 99% defect rates).
- Current T2I models exhibit significant and sometimes decreasing fairness in human image synthesis across gender, race, and age, indicating that quality improvements do not always correlate with reduced biases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.