Reproduce “UEval: A Benchmark for Unified Multimodal Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bo Li, Yida Yin, Wenhao Chai, Xingyu Fu, Zhuang Liu
Princeton University researchers developed UEval, a benchmark featuring 1,000 expert-curated questions and a novel rubric-based MLLM evaluation system, to assess models generating both images and text simultaneously. The benchmark revealed that frontier models achieved an average score of 66.4/100, significantly outperforming open-source counterparts while uncovering challenges in multi-step planning and image generation quality.
Key insights
- Unified multimodal models, particularly open-source ones, exhibit substantial weaknesses in integrated image and text generation, especially regarding image quality and maintaining visual consistency across multi-step sequences.
- Explicit reasoning (Chain-of-Thought) significantly improves the quality of generated visual outputs for capable models, highlighting its crucial role in guiding multimodal generation.
- The rubric-based MLLM-as-a-judge system, refined by human experts, offers a scalable, reliable, and diagnostically rich method for evaluating open-ended multimodal generation, showing strong alignment with human judgment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.