Reproduce “Characterization and Mitigation of Training Instabilities in Microscaling Formats” on OpenResearch

About this paper

Huangyuan Su, Mujin Kwun, Stephanie Gil, Sham Kakade, Nikhil Anand

The authors assess nearly 1,000 from-scratch language-model runs using MX and investigate instability. MX shares scales across blocks; they link quantization bias involving layer-normalization weights and activations to failures. Higher-precision activations or forward-only quantization mitigate instability. E4M3 weights with bfloat16 activations match baseline validation loss at 0.31B parameters, but software emulation establishes neither hardware speedups nor stability at larger scales.

Key insights

• Training instabilities in MX formats are driven by a "multiplicative gradient bias" stemming from specific quantization behaviors, rather than solely stochastic optimization dynamics.
• The root cause of this bias is the clamping of tightly clustered values within shared-scale quantization blocks, particularly impacting Layer Normalization affine parameters and, to a lesser extent, activations.
• Training divergence occurs when the operator norm of this quantization noise approaches a critical threshold (~2), indicating significant misalignment between low-precision and true gradients.

Summary by alphaXiv.