Reproduce “Lara Mini Base: Language Diversity as a Resource, Not a Trade-off” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Translated, Maria Pia Adinolfi, Isabelle Andrieu, Grazia Fiorella Anfuso, Melania Annunziata, Daniele Antonini, Marta Arena, Luca Barone and 198 more
Lara Mini base uses sparse experts to train a multilingual base language model. It has 7.2B total parameters, with 0.6B active per token, and was trained on 6T tokens across 45 languages. Its 57.0 Overall average ranks first among 36 open models with up to 2B active parameters, though it ranks fifth in code. In an ablation, replacing multilingual data raised English by 0.3 points but lowered multilingual scores by 2.7.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.