Reproduce “Parrot: Multilingual Visual Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo and 3 more
The PARROT framework enhances Multimodal Large Language Models (MLLMs) by enabling language-specific visual token alignment through a Mixture-of-Experts module, significantly improving non-English performance with minimal training data. Complementing this, the Massive Multilingual Multimodal Benchmark (MMMB) offers a standardized, diverse platform for evaluating MLLMs across six languages.
Key insights
- Visual features from English-centric vision encoders are poorly aligned with non-English textual inputs, necessitating language-specific adaptations.
- Textually guided Mixture-of-Experts (MoE) modules can effectively adapt English-biased visual embeddings into language-specific representations using minimal additional non-English data.
- Comprehensive and rigorously constructed benchmarks are crucial for objectively evaluating and advancing multilingual multimodal capabilities in MLLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.