Reproduce “Retrieval Capacity of Self-Attention Under Competition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Timur Mudarisov, Mikhail Burtsev, Radu State
The authors measure how much attention a model can restrict while preserving predictions. They retain top-ranked tokens with weights unchanged at each operation and assess loss increase. For Llama-3-8B on WikiText-103, the estimated size at 5% relative NLL tolerance rises from 10 tokens at context 256 to 66 at 2048. These are smallest tested passing sizes, not a universal scaling law.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.