Reproduce “Raven: High-Recall Sequence Modeling with Sparse Memory Routing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arshia Afzal, Aviv Bick, Eric P. Xing, Volkan Cevher, Albert Gu
Raven is a fixed-memory recurrent model that selectively writes to learned memory slots and uses decay to control how long their contents persist. Its router updates only selected slots, leaving the others unchanged. In NIAH-1 retrieval, Raven’s 400M-parameter model reaches 91.4% accuracy at 32K tokens, 16 times its training context; this does not establish general long-context recall.
Key insights
- Explicitly separating the decision of where information is written (routing) from how long it persists (selective decay) enables fine-grained, content-aware memory management.
- Raven naturally develops non-uniform Effective Sequence Lengths across its memory slots, allowing some slots to specialize in preserving long-lived information (e.g., 'retrieval slots').
- Increasing memory slot granularity (more slots for a fixed budget) improves retrieval quality, indicating the importance of finer-grained routing for memory organization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.