Reproduce “MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures” on OpenResearch

About this paper

Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue, Fuzhao Xue, Zian Zheng and 5 more

Researchers developed MixEval-X, a unified evaluation framework for "any-to-any" multi-modal AI models, which constructs dynamic benchmarks from real-world web queries to assess performance across diverse input-output modalities. This framework, developed at institutions including the National University of Singapore, demonstrated high correlations with user-facing leaderboards for multi-modal understanding tasks while highlighting inconsistencies between model judges and human preferences for multi-modal generation.

Key insights

• MixEval-X's sub-benchmarks closely align with real-world web query distributions, demonstrating the effectiveness of its pipelines in mitigating query bias and creating representative evaluations.
• Performance in specific multi-modal understanding tasks (e.g., Video2Text, Text2Audio) is significantly influenced by factors such as input resolution limits and model architecture, indicating areas requiring further research like long video understanding.
• Model-as-judge evaluations for open-ended multi-modal generation tasks exhibit low correlation (averaging 78%) with human preferences, despite high correlation among different model judges (85-95%), highlighting a need for more robust, human-aligned grading methods.

Summary by alphaXiv.