Reproduce “Mitigating Catastrophic Forgetting in Continual Learning through Model Growth” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ege Süalp, Mina Rezaei
Model growth strategies, specifically transformer stacking, are explored as a method to mitigate catastrophic forgetting in Large Language Models (LLMs) during continual fine-tuning. The study found that growth-based LLMs showed modest improvements in retaining reasoning and reading comprehension abilities, but exhibited a trade-off by being less effective at reducing social biases compared to conventionally trained LLMs.
Key insights
- Growth-based pre-training can lead to a more robust foundational knowledge representation, offering some resilience against catastrophic forgetting in cognitive tasks like reasoning and reading comprehension.
- The benefits of growth-based models in mitigating forgetting come with a complex trade-off: these models showed less adaptability in reducing social biases during continual learning compared to conventionally trained models.
- Optimal fine-tuning strategies, including learning rate schedulers and regularization, are critical for healthier training and can significantly impact performance, especially for growth-based architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.