Reproduce “Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jean Kaddour
Latest Weight Averaging (LAWA) reduces deep learning training time by averaging weights from the 'k' most recent checkpoints. This method enabled a ResNet50 model to reach 66% ImageNet accuracy 40 epochs (~68 GPU hours) faster and a RoBERTa-Base model to achieve its best WikiText-103 validation loss 45 epochs (~30 GPU hours) earlier.
Key insights
- Weight averaging can be repurposed to accelerate training convergence during the middle phase, rather than solely for improving generalization at the end of training.
- Latest Weight Averaging (LAWA) is fairly insensitive to the choice of 'k' (the number of latest checkpoints averaged) within a reasonable range (e.g., 5-16), but averaging too many or applying it too early in training can degrade performance.
- Uniform averaging coefficients for the latest checkpoints tend to perform slightly better than exponentially decaying coefficients.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.