Reproduce “MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianhao Li, Shangjie Li, Binbin Xie, Deyi Xiong, Baosong Yang
Researchers from Alibaba Group and academic institutions developed MoE-CT, a novel architecture for Large Language Models that enhances multilingual capabilities while providing superior resistance to catastrophic forgetting. This method dramatically reduces the amount of original training data required, needing over 95% less data compared to conventional continual training methods.
Key insights
- Freezing most of the pre-trained LLM's parameters, including original FFNs, effectively safeguards previously acquired knowledge against catastrophic forgetting.
- The Mixture of Experts (MoE) architecture successfully segregates the learning of original language proficiency from new multilingual knowledge acquisition.
- A selective training approach, updating only new expert networks and a fusion module, allows for robust multilingual expansion with substantially reduced reliance on original language data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.