Reproduce “Kwai Summary Attention V2 Technical Report” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
OneRec Team, Chenglong Chu, Guohong Mu, Guowang Zhang, Hao Peng, Hui Wang, Jian Liang, Jiangxia Cao and 25 more
KSA V2 uses Summary tokens both to compress distant-history key-value memory and to route sparse attention to selected original-text blocks. It preserves Summary representations between layers while compressing their stored keys and values. On 1.9B, it scores 72.39% accuracy across six RULER context lengths and 65.01% at 128K, close to Hybrid KSA; it evaluates retrieval through 128K, not broader tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.