Reproduce “BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhewen Shen, Aditya Joshi, Ruey-Cheng Chen
BAMBINO-LM introduces a human-inspired, two-phase continual pre-training strategy for small language models, combining causal language modeling with PPO-based feedback to facilitate second language acquisition. The method improved average Italian task scores by 0.1197, demonstrating effective bilingual learning with an anticipated, contained degradation in specific first language capabilities.
Key insights
- Interleaving standard language modeling with PPO-based perplexity feedback from a fluent "parent" model effectively guides a small LM in acquiring a second language.
- The observed degradation in specific first language (English BLiMP) capabilities, while general comprehension (English GLUE) remains stable, parallels phenomena in human second language acquisition.
- An alternating run strategy for CLM and PPO phases is crucial for successful second language acquisition, suggesting that frequent, interleaved interactions are beneficial.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.