Reproduce “Quasi Linear Kernel Attention with Infinite Capacity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicolaj Rux, Johannes Hertrich, Sebastian Neumayer
The authors develop additive-kernel attention that sorts keys and uses prefix sums to compute coordinatewise weighted values, avoiding query–key comparisons. They prove additive Laplace and bump kernels have infinite capacity, defined by attention matrices approaching identity, with quasi-linear computation. On one RTX 5090, additive-Riesz takes 600.6 ms versus FlashAttention’s 3,442 ms at length 131,072, using fp32 versus fp16.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.