Reproduce “Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gregor Geigle, Radu Timofte, Goran Glavaš
The paper introduces Babel-ImageNet, a massively multilingual zero-shot image classification benchmark covering 100 languages, derived from WordNet and BabelNet. This new benchmark reveals substantial performance disparities in existing vision-and-language models across different language resource levels and offers a scalable evaluation tool for multilingual model development.
Key insights
- Current multilingual VL models exhibit substantial performance drops for high-resource languages compared to English, and further drastic degradation for low- and mid-resource languages.
- English ZS-IC performance is not a consistent predictor of multilingual proficiency, highlighting a "curse of multilinguality" where model capacity struggles to maintain quality across many languages.
- Babel-ImageNet ZS-IC accuracy strongly correlates with text-to-image retrieval performance on established datasets, validating its use as a proxy for multilingual VL representation quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.