Reproduce “MEF: A Systematic Evaluation Framework for Text-to-Image Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaojing Dong, Weilin Huang, Liang Li, Yiying Li, Shu Liu, Tongtong Ou, Shuang Ouyang, Yu Tian and 1 more
ByteDance Seed developed MEF, a systematic evaluation framework for text-to-image models, integrating ELO-based competitive ranking with multi-dimensional MOS scoring on a scenario-aware benchmark called Magic-Bench-377. The framework quantitatively analyzes dimension contributions to user satisfaction and characterizes the performance of leading T2I models.
Key insights
- Prompt Following emerged as the most critical determinant of user selection and satisfaction across various groups and application scenarios in text-to-image generation.
- Expert evaluators exhibit greater sensitivity to subtle model performance differences, resulting in a wider dispersion of ELO scores compared to general users.
- Population preferences (e.g., designers vs. general users) and application-scenario preferences (e.g., Art vs. Functional Design) distinctly influence the perceived importance of different evaluation dimensions.
- Current T2I models consistently show limitations in complex relational reasoning, quantity representation, and advanced text expression forms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.