Reproduce “RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world Scenarios” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fei Zhao, Chengqiang Lu, Yufan Shen, Qimeng Wang, Yicheng Qian, Haoxin Zhang, Yan Gao, Yi Wu and 4 more
Nanjing University, Zhejiang University, and Xiaohongshu Inc. introduce RealBench, the first Chinese multi-image understanding benchmark, constructed from real user-generated content to evaluate multimodal large language models in authentic, complex scenarios. The benchmark demonstrates that current MLLMs, including leading closed-source models, face considerable challenges in cross-image reasoning and integration, particularly with an increasing number of images.
Key insights
- State-of-the-art MLLMs, including leading closed-source models, exhibit significant limitations in understanding complex multi-image Chinese content, especially in reasoning tasks.
- Model performance consistently degrades as task complexity increases and when more images need to be processed or integrated.
- A substantial performance gap of approximately 71.8% exists between open-source and closed-source MLLMs on RealBench, indicating significant room for improvement in open-source alternatives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.