Reproduce “Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zicheng Zhang, Tengchuan Kou, Shushi Wang, Chunyi Li, Wei Sun, Wei Wang, Xiaoyu Li, Zongyu Wang and 4 more
Researchers from Shanghai Jiao Tong University and Meituan introduced Q-Eval-100K, the largest human-annotated dataset featuring 100K text-to-vision instances with disentangled scores for visual quality and text alignment. They also developed Q-Eval-Score, an LMM-based model that leverages this dataset to provide accurate, decoupled evaluations for both images and videos, outperforming existing state-of-the-art metrics.
Key insights
- Generative models show varied performance, with visual quality consistently scoring lower than text-prompt alignment, indicating a greater challenge in producing aesthetically pleasing content.
- Large-scale, high-quality human annotations with disentangled scores are essential for effectively training LMMs to act as sophisticated content evaluators.
- LMMs can be guided through human-like reasoning processes using structured prompts ("Context-Prompt") and strategies for complex inputs ("Vague-to-Specific") to improve evaluation accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.