Reproduce “OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks” on OpenResearch

About this paper

Jiayu Wang, Yang Jiao, Yue Yu, Tianwen Qian, Shaoxiang Chen, Jingjing Chen, Yu-Gang Jiang

OmniGenBench introduces a comprehensive evaluation framework for Large Multimodal Models, encompassing 57 diverse sub-tasks that assess both perception-centric and cognition-centric instruction-following capabilities. The benchmark's evaluation of mainstream models demonstrates that GPT-4o-Native achieves state-of-the-art performance, particularly in complex reasoning and identity preservation, while also highlighting shared limitations across models in handling highly specialized professional knowledge.

Key insights

• OmniGenBench categorizes generative tasks into perception-centric (direct visual translation) and cognition-centric (requiring reasoning and knowledge) to comprehensively evaluate LMMs.
• A dual-mode evaluation framework combines automated scene parsing for perception tasks and an "MLLM-as-a-Judge" paradigm using Gemini-2.5-Pro for cognition tasks, ensuring robust and reproducible assessment.
• The OmniScore metric, prioritizing instruction adherence (80%) over realism (10%) and aesthetics (10%), provides a holistic measure validated by high agreement rates (0.84-0.96) with human judgment.

Summary by alphaXiv.