Reproduce “Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization” on OpenResearch

About this paper

Hanzhang Wang, Tianqi Shen, Zonglin Liu, Junze He, Difan Zou, Ziye Ma

The authors introduce LNWA, a family of averaging kernels balancing temporal lag and spatial noise in pretrained models quantized after training. Its coefficient trades less lag for more noise; the preferred balance can depend on precision. For OPT-125M under GPTQ, correlation across six endpoints shifts from −0.716 at 4-bit to +0.756 at 2-bit, an association, not causal or universal evidence.

Key insights

• Distributing weight across more checkpoints reduces uncorrelated noise but increases lag behind recent optimization progress; neither benefit can be improved without a trade-off.
• The cost of retained noise matters more as quantization precision falls, while at higher precision the benefit of preserving recent progress can carry greater weight.
• The theory relies on assumptions about checkpoint-noise correlations and a simplified quantization model, so its guidance is not a universal guarantee for practical quantization setups.

Summary by alphaXiv.