Reproduce “Hierarchical multimodal transformers for Multi-Page DocVQA” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny
Researchers at the Computer Vision Center (CVC) at UAB developed MP-DocVQA, a large-scale dataset for multi-page Document Visual Question Answering, and introduced Hi-VT5, a hierarchical multimodal Transformer. Hi-VT5 achieves 48.28% answering accuracy and 79.23% answer page identification accuracy on the MP-DocVQA dataset by efficiently processing long documents through page-level information summarization, surpassing existing baselines.
Key insights
- A hierarchical approach using learnable `[PAGE]` tokens for page summarization effectively bypasses the quadratic complexity limitation of traditional Transformers for long multi-page documents.
- A hierarchical layout-aware de-noising pre-training task is crucial for effectively training the `[PAGE]` tokens to embed relevant information from individual pages.
- The combination of textual, 2D positional, and visual features is essential for comprehensive multimodal understanding in multi-page document VQA, with each modality contributing to overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.