Reproduce “CPI-Bench: A Comprehensive, Practical and Intelligent Benchmark for Real-World Image Editing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qinye Zhou, Jun Zheng, Yongchao Du, Yuan Wang, Zhengrui Chen, Zuan Gao, Taihang Hu, Chao Lin and 12 more

Researchers from Alibaba Group developed CPI-Bench, a comprehensive benchmark for evaluating real-world image editing models that includes multi-image editing, practical application scenarios, and reasoning-based tasks. The benchmark achieves significantly higher performance differentiation among models and demonstrates a strong alignment with human perceptual preferences compared to existing evaluation systems.

Key insights

  • CPI-Bench demonstrated significantly higher performance variance among image editing models compared to traditional benchmarks, effectively highlighting differences in complex scenarios.
  • Closed-source models consistently outperformed open-source models across multi-image, real-world application, and reasoning-based tasks, indicating critical capability gaps in open-source solutions.
  • The benchmark's evaluation results showed strong alignment with human preferences, as evidenced by high Spearman correlation coefficients and low Mean Absolute Error when compared to the Arena Image Edit Leaderboard.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.