Reproduce “DocFormer: End-to-End Transformer for Document Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, R. Manmatha

DocFormer introduces an end-to-end trainable multi-modal Transformer architecture that deeply integrates text, spatial, and raw visual features for Visual Document Understanding (VDU). The model achieves state-of-the-art performance across various VDU benchmarks and often surpasses larger models in efficiency, leveraging novel multi-modal self-attention and specific unsupervised pre-training tasks for robust information extraction.

Key insights

  • Sharing spatial weights within the multi-modal self-attention layer effectively facilitates cross-modality feature correlation and enhances performance while reducing parameters.
  • Unsupervised pre-training tasks like Multi-Modal Masked Language Modeling (without masking visual regions for text) and Learn To Reconstruct (image reconstruction) effectively force multi-modal interaction and collaboration between visual and language features.
  • A streamlined, end-to-end architecture leveraging a plain ResNet50 for visual features can outperform more complex models that rely on external pre-trained object detectors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.