Reproduce “Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
The authors analyze how depth affects loss reduction in deep linear networks trained by population gradient flow on squared-loss tasks with power-law spectra. Depth changes optimization dynamics, not the predictor class. For difficult tasks with spectrum exponent p<1, the predicted loss exponent rises with depth; at p=1 only a logarithmic factor changes, and for p>1 the exponent is unchanged.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.