Reproduce “Recursive Sparse Attention: Million-Token Length Extrapolation at Logarithmic Cost” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruitian Wang, Xinyu Yang, Minshen Zhang, Hao Zhang
RecSA extends a language model’s usable context through recursive search over learned summaries, without training on documents as long as the target context. It selects candidate groups hierarchically and assigns candidates local positions. On the authors’ RULER reimplementation, Qwen3-4B scores 60.5 at 1M tokens versus 40.0 for Dense + DCA, supporting extrapolation on this model and setup, not general long-context performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.