Reproduce “PACT: Parameterized Clipping Activation for Quantized Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan
IBM Research AI introduces Parameterized Clipping Activation (PACT), a method enabling 4-bit precision for both weights and activations in deep neural networks like ResNet50 with less than 1% accuracy degradation. This technique dynamically learns optimal activation clipping ranges during training, leading to substantial hardware efficiency gains, including a 4.5x inference speedup in bandwidth-constrained systems due to improved on-chip memory utilization.
Key insights
- Learning the optimal clipping range for activations through back-propagation allows for a dynamic balance between dynamic range and quantization error, crucial for maintaining accuracy at ultra-low bit-precisions.
- An L2-regularizer applied to the clipping parameter 'α' is essential for smooth optimization and preventing the clipping range from becoming excessively large or getting stuck in suboptimal local minima.
- Layer-wise parameterization of 'α' provides the best trade-off between model accuracy and hardware implementation complexity, simplifying accelerator design.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.