Reproduce “Attention Residuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang and 29 more
AttnRes changes residual connections by letting each layer softmax-weight earlier representations using a learned query, rather than uniformly accumulating them. Block AttnRes compresses sources into block summaries. At 5.6 PFLOP/s-days, the fitted validation-loss curve reaches 1.692 versus 1.714 for baseline, a 1.25× compute-equivalent comparison, not measured speedup. In Kimi Linear, scores rose on 14 of 15 tasks and tied on one.
Key insights
- A formal duality exists between depth-wise accumulation in residual networks and sequential recurrence in RNNs, suggesting that content-dependent attention can similarly enhance information flow across network depth.
- Replacing fixed residual aggregation with learnable softmax attention effectively mitigates the 'PreNorm dilution' problem, leading to bounded hidden-state magnitudes and more uniform gradient distributions across layers.
- Optimal architectural scaling shifts towards deeper, narrower networks when using AttnRes, indicating its ability to better leverage increased model depth under fixed compute and parameter budgets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.