Reproduce “Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yixiao Qian, Yaxin Zhang, Song Chen, Jiaxu Liu, Shengze Cai, Chao Xu
Dual DeltaNet (DDN) adds a value-side read to Kimi Delta Attention, using selected value features to adjust how content is represented across addresses in a fixed-size recurrent state. On 1B parameters, it averages 74.52% across 12 RULER needle settings, versus 69.40% for GDN-2 and 64.27% for KDA. On six real-world retrieval tasks, its 1B average is 31.20%, below GDN-2's 31.80%.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.