Reproduce “Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinyu Liu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang
Unison is a new benchmark for evaluating Unified Multimodal Models (UMMs) by assessing the synergistic interplay between their understanding and generation capabilities across four dimensions. Evaluation on this benchmark reveals that while closed-source models perform better overall, even high-performing UMMs frequently struggle with genuine cross-modal consistency and iterative self-refinement in complex tasks.
Key insights
- Internal consistency, where a UMM's semantic encoding aligns across both understanding and generation, is a fundamental prerequisite for achieving high overall performance in unified tasks.
- Multimodal synergy is observed, demonstrating that understanding and generation capabilities can mutually enhance each other, with reasoning guiding contextually accurate generation and generated visual contexts sometimes aiding understanding.
- UMMs possess a latent ability for autonomous self-improvement through iterative internal feedback loops, allowing them to refine outputs and extract value even from initially suboptimal internal states.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.