Reproduce “Low-bit Quantization of Neural Networks for Efficient Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yoni Choukroun, Eli Kravchik, Fan Yang, Pavel Kisilev
Research from Huawei Technologies Co. introduces a framework for post-training, 4-bit integer (INT4) quantization of deep neural networks, enabling efficient inference on resource-constrained hardware. This framework achieves a mean top-1 accuracy degradation of only 3% and top-5 degradation of 1.7% across diverse architectures on ImageNet, without requiring full retraining or mixed-precision hardware.
Key insights
- Kernel-wise quantization for convolutional layer weights is essential for preventing catastrophic accuracy collapse when quantizing to INT4 precision.
- Approximating key layers with a sum of multiple low-precision tensors effectively recovers accuracy for sensitive parts of the network while maintaining compatibility with uniform low-bit hardware.
- A data-driven, differentiable post-quantization refinement of scaling factors on a small calibration set provides a practical and highly effective method for closing the accuracy gap in aggressively quantized models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.