Reproduce “Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jary Pomponi, Simone Scardapane, Aurelio Uncini
Structured Ensembles proposes a technique to reduce the memory footprint of deep neural network ensembles by extracting diverse, functionally important sub-networks from a single, untrained parent network. The method also extends to continual learning, effectively mitigating catastrophic forgetting with negligible memory overhead and outperforming existing architectural strategies.
Key insights
- Diverse, high-performing sub-networks can be extracted directly from an untrained, over-parameterized neural network using differentiable scaling and gradient-based importance.
- A unified approach based on extracting and protecting specialized sub-networks can effectively address both the memory footprint of deep ensembles and catastrophic forgetting in continual learning.
- Explicit diversity regularization during sub-network extraction is crucial for ensuring effective ensemble performance and predictive calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.