Reproduce “Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao, Lu Xiang, Chengqing Zong, Yu Zhou

M4Doc enhances Document Image Machine Translation (DIMT) quality and generalization by transferring multimodal knowledge from large language models (MLLMs) to a smaller, efficient model. The framework achieves state-of-the-art translation performance and robust adaptability across diverse document types, while significantly reducing inference computational overhead by 44.1%.

Key insights

  • Distilling multimodal knowledge from MLLMs into a smaller, image-only encoder via an alignment loss effectively enhances the student model's generalization capabilities for DIMT tasks.
  • MLLMs pre-trained on document images (e.g., Vary-toy, Textmonkey) function as more effective teachers for DIMT compared to those primarily trained on natural images, influencing student performance.
  • The `L_align` component and the use of both image and source text as input to the MLLM during training are crucial for generating robust mix-modality representations and enabling effective knowledge transfer.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.