Reproduce “Code-Switching Curriculum Learning for Multilingual Transfer in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haneul Yoo, Cheonbok Park, Sangdoo Yun, Alice Oh, Hwaran Lee
The paper introduces Code-Switching Curriculum Learning (CSCL), a multi-stage pre-training method for LLMs that mimics human second language acquisition by progressively reducing code-switching in training data. This approach substantially improves target language performance, mitigates catastrophic forgetting of the source language, and enhances safety across various languages and models.
Key insights
- Mimicking the progressive stages of human second language acquisition through a structured code-switching curriculum effectively optimizes LLM transfer and preserves source language capabilities.
- Explicit token- and sentence-level code-switching provides crucial cross-lingual alignment cues, enabling more consistent knowledge and reasoning across languages.
- The proposed method is efficient and robust in low-resource settings, as it reduces reliance on extensive human-curated monolingual data by leveraging synthetically generated code-switching data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.