Reproduce “Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziteng Wang, Jianqiao Lu, Xinyu Ma, Jintao Zhang, Jun Zhu, Yiyuan Ma, Jianfei Chen
HiRes separates carried residual width from layer-computation width in Transformers. It carries a wider state between groups of ordinary PreNorm layers, which still compute at width d, using block-boundary projections and RMSNorm. In five model-size comparisons, its fitted training FLOPs to validation loss 3.00 are lower; the vanilla baseline requires 1.30× as much, but this is not a wall-clock speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.