Reproduce “Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation” on OpenResearch

About this paper

Isabela Albuquerque, Ira Ktena, Olivia Wiles, Ivana Kajić, Amal Rannen-Triki, Cristina Vasconcelos, Aida Nematzadeh

Google DeepMind researchers developed a principled framework for evaluating text-to-image model diversity through attribute-conditional human evaluation, improving assessment accuracy to over 80%. This framework benchmarked leading T2I models, identifying Imagen 3 and Flux 1.1 as top performers and providing insights into the alignment of automated diversity metrics with human perception.

Key insights

• Human perception of diversity is significantly more accurate and consistent when the specific attribute of variation is explicitly defined for evaluators.
• While newer T2I models like Imagen 3 and Flux 1.1 exhibit improved attribute-specific diversity, "mode collapse," where models produce overly similar images, persists as a challenge.
• The performance of automated diversity metrics, such as the Vendi Score, is strongly influenced by the choice of embedding space and shows higher agreement with human judgments for larger diversity differences.
• Large vision-language models can potentially serve as effective automated diversity evaluators, demonstrating high accuracy on controlled diversity tasks.

Summary by alphaXiv.