Reproduce “(How) Learning Rates Regulate Catastrophic Overtraining” on OpenResearch

About this paper

Mark Rofin, Aditya Varre, Nicolas Flammarion

Research from EPFL provides a mechanistic explanation for catastrophic overtraining in Large Language Models, showing that pretraining learning rate decay increases model sharpness, which then amplifies feature drift and forgetting during supervised finetuning. The study offers practical recommendations for controlling finetuning learning rates and pretraining sharpness to mitigate this effect.

Key insights

• Lower finetuning learning rates implicitly regularize SFT, effectively preserving pretrained features by reducing feature drift without impeding in-distribution performance.
• Model sharpness, which progressively increases during pretraining (especially with learning rate decay), is a primary driver amplifying feature drift and subsequent forgetting during finetuning.
• The optimization dynamics show that increasing the finetuning LR disproportionately amplifies feature drift compared to its effect on SFT training loss reduction per step, leading to greater accumulated drift over time.

Summary by alphaXiv.