Reproduce “CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models” on OpenResearch

About this paper

Pavan Kalyan, Shubhra Mishra, Satya Lokam, Navin Goyal

The paper introduces CurLL, a comprehensive framework and 23.4 billion-token synthetic dataset designed to evaluate continual learning in language models. It grounds learning progression in human developmental curricula (ages 5-10) and models explicit skill dependencies via a skill graph, enabling fine-grained analysis of skill acquisition, forgetting, and knowledge transfer.

Key insights

• Continual training can lead to better generalization to new, unseen skills (later stages) compared to joint training, but it comes with a trade-off of increased catastrophic forgetting of earlier skills.
• The explicit skill graph effectively diagnoses forgetting patterns, showing that skills less foundational for subsequent learning (fewer outgoing edges) are more vulnerable to catastrophic forgetting during continual training.
• Exposing models to a diverse range of skills concurrently (joint training) improves overall performance and retention of earlier skills compared to independent training or purely sequential continual learning.

Summary by alphaXiv.