Reproduce “Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Minglai Yang, Xinyan Velocity Yu, Pengyuan Li, Xinyu Guo, Zhenting Qi, Konwoo Kim, Longtian Ye, Xiaolong Luo and 17 more

Researchers introduced DR.DOCBENCH, a benchmark designed to evaluate document parsing capabilities of vision-language models on expert-level and challenging documents across 52 subject domains. Evaluations revealed current models struggle significantly with complex layouts, specialized content like MusicXML and chemical structures, and long-document parsing, demonstrating persistent performance gaps in challenging subjects and component types.

Key insights

  • Current vision-language models demonstrate significant performance degradation on expert-level and structurally complex documents, particularly in subjects like 'REFERENCE', 'DESIGN', and 'MEDICAL'.
  • Many parsing errors originate from layout and visual-structure ambiguity (e.g., multi-column pages, rotated text, colored/borderless tables) rather than just plain text recognition.
  • Increasing the multi-page context length does not consistently improve long-document parsing, with reading order performance often worsening as window size grows.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.