Reproduce “Functional Degeneracy in Neural Networks: Measurement and Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maria Matveev, Pascal Esser, Ayush Bharadwaj, Lucius Bushnaq, Gitta Kutyniok
The authors define behavioral recovery rank, the smallest number of leading eigendirections of a behavioral Hessian needed for a projected checkpoint to meet a performance criterion. The Hessian compares predictions with a trained checkpoint on a probe set. In teacher–student regression, growth slows after width 10, while no-retraining pruning counts keep rising; refitting narrows the gap.
Key insights
- In the synthetic teacher–student task, recovery rank grows more slowly once student width reaches the teacher’s width, whereas it keeps increasing with width on MNIST within the tested range.
- Behaviorally important directions can combine many weights and neurons, so conventional pruning criteria may not align with the redundancy revealed by the functional geometry.
- Refitting the output layer improves structural pruning in the synthetic setting, indicating that pruning’s gap from recovery rank partly reflects limited compensation after removal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.