Reproduce “Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adam Ousherovitch, Yixin Wang
The authors introduce Terminal Shrinkage Averaging (TSA), which blends the final checkpoint with an average of checkpoints, separating the returned model from the learning-rate schedule. In five depth-12 NanoChat runs, raising the terminal learning-rate floor from 5% to 15% worsened raw-checkpoint validation BPB by 0.003757 and improved TSA BPB by 0.000844; this result does not establish transfer beyond NanoChat settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.