Reproduce “SWALP : Stochastic Weight Averaging in Low-Precision Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Christopher De Sa
SWALP extends stochastic weight averaging to low-precision training by averaging model snapshots along a quantized SGD path. Their average can lie between representable values, though the usual averaging accumulator remains high precision. On VGG-16/CIFAR-100 with 8-bit small-block BFP, test error falls from 29.59% for low-precision SGD to 26.65%, below full-precision SGD’s 27.23% but above full-precision SWA’s 25.93%.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.