Reproduce “Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression” on OpenResearch

About this paper

Dachi Kurtskhalia

Researchers at Impel.ai investigate the behavioral effects of serving-time weight compression on large language models, revealing that 4-bit quantization reduces output diversity in smaller 8B models and causes stylistic drift in larger 14B and 32B models. The study clarifies that quantization primarily amplifies determinism on modal answers rather than systematic bias, finding that increased token-level noise can paradoxically lead to less semantically diverse outputs.

Key insights

• Quantization effects are scale-dependent: smaller 8B models show reduced content diversity, while larger 14B and 32B models exhibit stylistic drift (e.g., increased em-dash rate) without significant content distribution changes.
• Quantization amplifies determinism or concentration on modal answers rather than systematically increasing pre-existing stereotypes or biases.
• There is a dissociation between token-level noise and semantic diversity: increased token-level entropy (flatter distributions) can coincide with decreased semantic diversity, suggesting a collapse of meaningful alternatives.

Summary by alphaXiv.