Reproduce “SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks” on OpenResearch

About this paper

Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W. Keckler and 1 more

Researchers from NVIDIA, MIT, UC-Berkeley, and Stanford University developed SCNN, a hardware accelerator designed for Convolutional Neural Networks that efficiently exploits the sparsity found in both network weights and activations. This system achieved an average speedup of 2.7x and an average energy efficiency improvement of 2.3x over dense CNN accelerators on benchmark networks like AlexNet, GoogLeNet, and VGGNet.

Key insights

• Combining weight sparsity from pruning and activation sparsity from ReLU can reduce required CNN computation by factors of 4 to 10 for typical layers.
• An accelerator specifically designed to operate on and maintain compressed-sparse data can achieve substantial performance and energy efficiency gains by eliminating zero-operand computations and reducing data movement.
• Intra-PE multiplier array underutilization (fragmentation) due to sparse data is a more critical performance bottleneck than inter-PE load imbalance, especially in later CNN layers with smaller working sets.

Summary by alphaXiv.