Reproduce “MT$^{3}$: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaopeng Feng, Yupu Liang, Shaosheng Cao, Jiayuan Su, Jiahan Ren, Zhe Xu, Yao Hu, Wenxuan Huang and 2 more

Researchers developed MT³, a multi-task reinforcement learning framework that significantly enhances Text Image Machine Translation (TIMT) using Multimodal Large Language Models. This framework achieves state-of-the-art results on the MIT-10M benchmark and strong generalization, while also introducing XHSPost, a new social media TIMT benchmark.

Key insights

  • Decomposing Text Image Machine Translation (TIMT) into explicit recognition, reasoning, and translation sub-tasks within a multi-task reinforcement learning framework significantly enhances overall performance and interpretability.
  • A "zero-start" RL approach, training directly from a general MLLM checkpoint, demonstrates superior learning efficiency and achieves a higher performance ceiling compared to models initialized with supervised fine-tuning.
  • The multi-mixed reward mechanism, combining format and task-specific continuous metrics, is critical for providing fine-grained feedback that leads to robust and holistically improved translation quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.