Reproduce “OCRGenBench: A Comprehensive Benchmark for Evaluating OCR Generative Capabilities” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peirong Zhang, Haowei Xu, Jiaxin Zhang, Xuhan Zheng, Guitao Xu, Yuyi Zhang, Junle Liu, Zhenhua Yang and 2 more

Researchers from South China University of Technology developed OCRGenBench, a comprehensive benchmark and evaluation metric for assessing generative models' OCR capabilities across 33 tasks and five text categories. The evaluation of 19 state-of-the-art models revealed significant deficiencies, particularly in document and handwriting tasks and precise pixel-level fidelity, with the top-performing model, Nano Banana Pro, achieving an OCRGenScore of 77.19 out of 100.

Key insights

  • A significant gap exists between models' ability to generate perceptually appealing content (high VIEScore) and their actual text accuracy (low AR), indicating insufficient text rendering capability even in T2I generation.
  • Models generally perform better on creative tasks like T2I generation and text editing, which prioritize perceptual quality, but struggle more with restorative tasks demanding pixel-level fidelity, often leading to hallucinations.
  • Unified generative models, while showing higher potential in closed-source implementations (e.g., Nano Banana Pro), lag significantly in open-source versions compared to specialized architectures, suggesting substantial engineering resources are currently required for high performance in unified models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.