Reproduce “NIPQ: Noise proxy-based Integrated Pseudo-Quantization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo, Eunhyeok Park

NIPQ, a noise proxy-based Pseudo-Quantization Training framework from POSTECH and Seoul National University, enables stable and accurate low-bit and mixed-precision quantization for both activations and weights. The method achieves state-of-the-art results across diverse vision and language tasks, improving ImageNet Top-1 accuracy for MobileNet-v2 at 4-bit to 71.52% and demonstrating significant gains on GLUE for BERT-base at 3-bit.

Key insights

  • Pseudo-Quantization Training (PQT) with a noise proxy provides a more stable training paradigm than STE-based methods, crucial for achieving high accuracy at very low bit-widths (sub-4-bit) for complex networks.
  • Integrating truncation and joint gradient-based optimization of all quantization parameters (bit-widths, truncation boundaries) within PQT enables efficient, automated mixed-precision allocation tailored to resource constraints.
  • NIPQ guides model training to converge to flatter loss surfaces, thereby enhancing the network's inherent robustness against noise and improving the stability of learned quantization parameters.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.