Reproduce “Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, Goran Glavaš

This research systematically investigates the optimal training strategies for massively multilingual Large Vision-Language Models, demonstrating that performance scales effectively to 100 languages without significant degradation and identifying effective data distributions for both pre-training and instruction-tuning. The work introduces a new benchmark for multilingual OCR and develops Centurio, a model that achieves state-of-the-art results across 56 languages.

Key insights

  • Scaling Large Vision-Language Models to 100 languages does not induce a "curse of multilinguality"; performance on existing languages, including English, is negligibly impacted while new languages benefit.
  • Optimal multilingual performance is achieved when non-English data constitutes 25-75% in instruction-tuning and approximately 50% in pre-training data, suggesting that exposure to a language is more crucial than its exact proportion.
  • Incorporating synthetic multilingual OCR data and fine-tuning the image encoder are effective for improving text-in-image understanding, though a notable performance gap persists for non-Latin scripts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.