Reproduce “FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zekun Shi, Alastair James Arthur Price
FreeSolo uses mean-key cell summaries to route queries to distant context, then reads original token keys and values from selected cells, alongside local attention. FS-L keeps a fixed pool; FS-B searches a growing archive. After 4k training, the 800M model retrieves a single needle at 256k, but this does not establish dense fact recall.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.