Reproduce “WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
An-Lan Wang, Jingqun Tang, Liao Lei, Hao Feng, Qi Liu, Xiang Fei, Jinghui Lu, Han Wang and 5 more
WildDoc evaluates multimodal document-understanding systems on more than 12,000 photographs of printed documents captured under changes in viewpoint, lighting, shadows, blur, and paper deformation. Across DocVQA, ChartQA, and TableVQA, all tested models deteriorate substantially relative to clean benchmark images; even the strongest reported system reaches 73.7% aggregate accuracy but only 55.0 consistency across four views, exposing a large gap between standard benchmark performance and reliable real-world use.
Key insights
- Image quality degradation is not the main reason for the observed model failures: human accuracy falls by only 0.9 percentage points on WildDoc, while PaddleOCR line accuracy declines by 20.2%.
- Wrinkles, creases, bends, and angled views are more damaging than screen moiré; for Qwen2.5-VL-72B, these factors reduce performance by roughly 17.6–21.1 points, compared with about 8.3 points for moiré.
- Clean benchmarks compress the differences among strong models, whereas WildDoc separates them more clearly: the top five models’ DocVQA standard deviation rises from 0.612 on the original data to 6.1 on WildDoc.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.