Reproduce “mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu and 5 more

DAMO Academy's mPLUG-DocOwl is a modularized Multimodal Large Language Model tailored for OCR-free document understanding, enhancing fine-grained text and layout comprehension in diverse documents. It exhibits improved instruction following on a new human-evaluated benchmark, LLMDoc, and achieves competitive performance on standard document understanding datasets.

Key insights

  • Specialized instruction tuning on diverse OCR-free document datasets can successfully adapt general MLLMs to comprehend fine-grained visual text and complex document layouts.
  • Modular fine-tuning, particularly of the visual abstractor and LoRA modules, efficiently integrates domain-specific knowledge into large pre-trained MLLMs while preserving general multimodal abilities.
  • Balancing specialized document understanding with general language and vision-language skills via a two-stage training strategy creates a versatile and capable model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.