Reproduce “LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding” on OpenResearch

About this paper

Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei

LayoutXLM extends multimodal pre-training for visually-rich document understanding to 53 languages, leveraging 30 million documents to jointly learn from text, layout, and image features. The model establishes new state-of-the-art results on the newly introduced XFUND benchmark across 8 languages for semantic entity recognition and relation extraction, demonstrating robust zero-shot transfer capabilities.

Key insights

• Jointly learning from text, layout, and image modalities is crucial for robust document understanding in multilingual contexts.
• Multimodal pre-training facilitates strong zero-shot transfer capabilities across languages, as the model can capture common layout invariances from English to unseen non-English documents.
• Leveraging character-level bounding boxes and a unigram language model based SentencePiece tokenizer helps unify multilingual multimodal inputs, mitigating language-specific word segmentation issues.

Summary by alphaXiv.