Reproduce “Nested Sparsity for Scalable Pretraining and Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianzhe Chu, Druv Pai, Yi Ma
NESTOR extends conditional computation to attention, routing tokens to columns that restrict both context and available experts. In language-model pretraining, its 500M-active-parameter model reached held-out loss of 2.758 nats versus 2.793 for MoE after 25B tokens. Fitted scaling curves estimate MoE needs 1.29× its training compute to match; results reach 1B active parameters and assess held-out loss, not downstream benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.