Reproduce “Towards Complex Document Understanding By Discrete Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, Tat-Seng Chua
Researchers from the National University of Singapore and 6Estates Pte Ltd introduce TAT-DQA, a novel multi-modal Document VQA dataset featuring real-world, multi-page financial reports that demand complex discrete reasoning. They propose MHST, a Multi-Head Strategy Transformer, which achieved 41.5 EM and 50.7 F1 on TAT-DQA, outperforming strong baselines and highlighting the crucial role of multi-modal information and tailored reasoning strategies.
Key insights
- Integrating visual and layout information with text (multi-modal approach) is critical for effective Document VQA, consistently outperforming text-only models on complex documents.
- Discrete numerical reasoning, particularly arithmetic operations across various document components, presents the most substantial challenge for current AI models in Document VQA.
- Explicitly recognizing and processing tabular data separately, along with effectively handling information across multiple pages, are essential yet underexplored areas that significantly impact model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.