Reproduce “PaLI: A Jointly-Scaled Multilingual Language-Image Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner and 21 more
Google Research's PaLI, a jointly-scaled multilingual language-image model, achieves high performance across diverse vision-language tasks, including 149.1 CIDEr on COCO Captioning and 84.3% accuracy on VQAv2. The model leverages a new 4-billion parameter vision transformer (ViT-e) and a 100-language WebLI dataset, indicating that scaling vision components offers a favorable return on investment for multimodal capabilities.
Key insights
- Empirical validation demonstrates that jointly scaling both vision and language components leads to superior multimodal performance.
- Scaling the visual component (ViT-G to ViT-e) provides a better return on investment in accuracy improvement per parameter compared to solely increasing language model capacity.
- Training with massively multilingual data not only boosts multilingual performance but also unexpectedly improves performance on English-only tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.