Reproduce “MSQ: Memory-Efficient Bit Sparsification Quantization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Seokho Han, Seoyeon Yoon, Jinhee Kim, Dongwei Wang, Kang Eun Jeon, Huanrui Yang, Jong Hwan Ko
MSQ (Memory-Efficient Bit Sparsification Quantization) introduces a training-efficient mixed-precision quantization method for deep neural networks, addressing the high memory and computational costs of previous bit-level approaches. It achieves up to an 8x reduction in trainable parameters and an 86% speed-up in training time while maintaining competitive accuracy across CNNs and Vision Transformers.
Key insights
- Accurately computing Least Significant Bits (LSBs) with proper quantization bin alignment and balanced gradient direction is crucial for effective bit-level sparsification through regularization.
- Directly regularizing LSBs from continuous weights eliminates the need for explicit bit-level trainable parameters, drastically reducing GPU memory consumption and overall training time for mixed-precision quantization.
- Integrating Hessian information into the pruning strategy allows for adaptive, layer-wise precision reduction, leading to a more robust and efficient mixed-precision scheme that balances compression and accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.