Reproduce “DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huanrui Yang, Wei Wen, Hai Li
Researchers at Duke University developed DeepHoyer regularizers, a new family of differentiable and scale-invariant sparsity measures for deep neural networks. These regularizers achieve higher compression rates and FLOPs reductions on models like LeNet, AlexNet, and ResNet across various datasets while maintaining or improving accuracy compared to existing pruning methods.
Key insights
- The Hoyer measure, adapted through squaring, offers a robust, scale-invariant, and almost everywhere differentiable metric for sparsity, bridging the practical and theoretical benefits of L1 and L0 norms.
- DeepHoyer regularizers induce an adaptive 'trimming effect' where smaller weights are more aggressively pushed towards zero than larger ones, facilitating more targeted and effective sparsity.
- The proposed regularizers effectively induce both fine-grained element-wise sparsity and hardware-friendly structural sparsity, enhancing flexibility for diverse model compression objectives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.