Reproduce “DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junqing Lin, Jingwei Sun, Guangzhong Sun
DeltaLog introduces a deferred materialization strategy for recurrent linear attention models, optimizing decoding by reducing frequent, dense recurrent-state write-backs to high-bandwidth memory. The approach achieves up to 1.86x kernel-level speedup and 1.20x end-to-end serving speedup, primarily by reducing recurrent-state write traffic by up to 7.83x across different architectures like GDN, KDA, and RWKV6.
Key insights
- The frequent, dense write-back of recurrent states is a memory-bound bottleneck that can be alleviated by deferring state materialization.
- Decomposing the live recurrent state into a dense base and a compact, bounded update log significantly reduces overall memory traffic for dense write operations.
- The optimal merge interval (M) balances the amortization of dense write-backs against the computational overhead of replaying deferred log entries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.