Reproduce “Logarithmic Pruning is All You Need” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Laurent Orseau, Marcus Hutter, Omar Rivasplata
This work by DeepMind provides stronger theoretical guarantees for the "pruning is all you need" conjecture, demonstrating that randomly initialized overparameterized networks require only a logarithmic factor (in all variables except depth) more neurons per target weight to contain a high-performing subnetwork, a significant improvement over previous polynomial bounds.
Key insights
- Target weights can be approximated with logarithmic efficiency (in precision $\epsilon_w$) by decomposing them into a sum of components, each implemented by a dedicated intermediate neuron pair.
- Sampling weights from a hyperbolic distribution, particularly when combined with strategies for approximating their products, dramatically reduces the number of intermediate neurons needed.
- Batch sampling enables the reuse of intermediate neurons across multiple target weights within a layer, further reducing the overall overparameterization required.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.