Reproduce “Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Datong Zhou, Blake Bordelon
The authors study how sliding residual computations across width and depth affects training dynamics. Under the paper’s scaling rules, a positive residual-width-to-hidden-channel ratio preserves stochastic residual variation while individual hidden features remain learnable at order-one scale. At matched 124,475,904 parameters, GPT-2 training loss was 3.588404 for one stage and 3.049850 for 24 stages, in a single-seed comparison.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.