Reproduce “Understanding Emergent Abilities of Language Models from the Loss Perspective” on OpenResearch

About this paper

Zhengxiao Du, Aohan Zeng, Yuxiao Dong, Jie Tang

Du and colleagues test whether pre-training loss tracks downstream performance. They find models at similar loss generally perform similarly under fixed training conditions. On MMLU, accuracy stayed near the 25% random baseline above loss 2.2, then rose below it. Similar thresholds appeared on C-Eval and two math tasks. These results apply to the tested setup, not universally.

Key insights

• Pre-training loss functions as a universal and more precise predictor of a language model's downstream task performance compared to model size or training compute.
• Emergent abilities are real phenomena, occurring when pre-training loss drops below specific, consistent thresholds, even when measured by continuous evaluation metrics.
• Tasks exhibit two distinct performance-vs-loss relationships: smooth improvement for some, and sudden 'emergence' after a loss threshold for others, like complex reasoning tasks.

Summary by alphaXiv.