Reproduce “MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhang Li, Zhibo Lin, Qiang Liu, Ziyang Zhang, Shuo Zhang, Zidun Guo, Jiajun Song, Jiarui Zhang and 2 more
MDPBench is introduced as the first systematic benchmark designed to evaluate document parsing models across 17 diverse languages and challenging real-world photographed conditions. The benchmark reveals that current models experience substantial performance degradation, averaging 17.8% on photographed documents and 14.0% on non-Latin scripts.
Key insights
- Current document parsing models, including state-of-the-art vision-language models, display notable performance drops when handling photographed documents (average 17.8% decline) and non-Latin scripts (average 14.0% decline).
- Specific error patterns, such as language drift, hallucinated spaces in unspaced languages (e.g., Thai), and incorrect reading order for right-to-left scripts (e.g., Arabic), highlight critical limitations in current model robustness and multilingual understanding.
- While layout detection shows relative insensitivity to language differences, text, formula, and table recognition components are highly susceptible to both script type and image degradation, indicating a need for more generalized recognition capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.