Reproduce “Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mana Sakai, Masaaki Imaizumi
The authors analyze learning-rate transfer from a smaller proxy to a wider shallow linear network using full-batch gradient descent. They compare the loss from reusing the proxy’s rate with the difference between separately optimized losses at the same horizon. Under endpoint spectral and search-interval conditions, transfer is fast for horizons growing slower than square-root width, not a neural-network result.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.