Reproduce “ReWY: Faster Backpropagation for Delta-Rule Attention - From 7 to 3 Matrix Multiplications” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Zhang, Zhixin Zhao, Weiwei Sun, Yitao Hu, Sheng Chen, Keqiu Li
ReWY simplifies the backward pass of chunk-parallel delta-rule attention without changing the recurrence or objective. It treats each chunk’s correction as a triangular residual problem and shares a reverse sensitivity across gradients, reducing the local WY graph from seven matrix products to three. On H100, GDN2 complete low-level backward is 1.7033× the frozen FLA reference under full saving, an operator—not model-wide—result.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.