Reproduce “Kronecker Delta Rule for Long Context Linear Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer, Sebastian Böck, Sepp Hochreiter
KronLSTM compresses long token histories into fixed-size memory and updates the scalar strength of a key–value association rather than rewriting a value vector. Its gated write separates forgetting, erasing, and writing. In extended-context RULER retrieval, it reaches 41.8% accuracy on S3 at 16k, versus 7.4% for GDN-2; results favor it on natural-text retrieval, but not every task or context length.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.