Reproduce “Stochastic activations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maria Lomeli, Matthijs Douze, Gergely Szilvasy, Loic Cabannes, Jade Copet, Sainbayar Sukhbaatar, Jason Weston, Gabriel Synnaeve and 2 more
The authors train language-model feed-forward layers with a stochastic mix of ReLU and SiLU, then use ReLU at inference to expose sparse computation. In an LM3B implementation, this approach measured a 1.65× speedup in total per-token generation time on one CPU core at 90% activation sparsity. Its average benchmark score was close to, but below, the SiLU reference; results are setup-specific.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.