Reproduce “Document Understanding Dataset and Evaluation (DUDE)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jordy Van Landeghem, Rubén Tito, Łukasz Borchmann, Michał Pietruszka, Paweł Józiak, Rafał Powalski, Dawid Jurkiewicz, Mickaël Coustaty and 5 more
The Document Understanding Dataset and Evaluation (DUDE) introduces a comprehensive benchmark for visually-rich, multi-page documents, demonstrating a substantial performance gap between current state-of-the-art models and human capabilities in document understanding. The dataset challenges models with diverse document types, complex question types, and comprehensive answer formats, aiming to drive research toward more practically applicable Document AI systems.
Key insights
- A substantial performance gap exists between current state-of-the-art models (46.06 ANLS for the best model) and human capabilities (74.76 ANLS) on complex, real-world document understanding.
- Existing models, including large language models, exhibit significant weaknesses in deeply integrating visual and layout information for reasoning, as well as performing arithmetic and multi-hop operations.
- The ability to process long input sequences is critical for understanding multi-page documents, highlighting the need for architectures that can effectively handle extensive contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.