Reproduce “History Replays Itself: Unleashing Long-Context Capability in Linear Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski and 5 more
NoteBank adds retrieval to recurrent language models by archiving selected text spans as history arrives, then selecting and replaying spans through the same model when a question comes. On six retrieval tasks at one million tokens, it raises GDN’s unweighted mean score from 1.71% to 70.33% with top-two replay. Results vary by backbone; storage and matching grow with history.
Key insights
- Separating evidence preservation from recurrent-state computation can recover long-range recall without changing pretrained backbone weights or architecture.
- Failures arise at multiple stages: relevant evidence may not be archived or retrieved, and replaying more text is not always helpful; oracle selection exposes routing headroom, especially for Mamba-2.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.