Reproduce “Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression” on OpenResearch

About this paper

Sara Makenali, Babak Rokh, Ali Azarpeyvand

Researchers at the University of Zanjan developed two integrated strategies for deep neural network compression, combining geometric median-based pruning and Additive Powers-of-Two quantization. Their methods reduced model size by up to ~16x and bit-operations by over ~126x, while often preserving or even improving classification accuracy, such as a 0.06% accuracy increase on ResNet-110 for CIFAR-10.

Key insights

• Integrating pruning and quantization through tailored simultaneous or sequential strategies can yield significantly higher compression ratios than individual methods, with minimal impact on accuracy.
• Geometric Median-based pruning effectively identifies and removes redundant filters, contributing to compression without substantial information loss.
• APoT quantization is particularly beneficial for hardware implementation, as it facilitates efficient bit-shift operations and enables low-bit precision representations for weights and activations.

Summary by alphaXiv.