Reproduce “Quantized Distributed Training of Large Models with Convergence Guarantees” on OpenResearch

About this paper

Ilia Markov, Adrian Vladu, Qi Guo, Dan Alistarh

QSDP (Quantized Fully-Sharded Data Parallel) introduces a method for quantizing both weights and gradients within the FSDP framework, offering theoretical convergence guarantees for optimization on quantized domains. Developed by researchers at IST Austria, CNRS & IRIF, and Max Planck Institute for Informatics, QSDP enables up to 2.25x speedups for 1.3B GPT models on 10 Gbps networks while preserving model accuracy.

Key insights

• The paper provides theoretical convergence guarantees for SGD where weights are restricted to a non-convex quantized domain, a unique contribution for training with quantized weights.
• Communication in FSDP is dominated by weight exchanges during both forward and backward passes, unlike traditional data-parallel training where gradients are the primary bottleneck.
• Using a 'bucketing' strategy, where large tensors are split and quantized independently, is crucial for preserving accuracy when applying weight quantization during training.

Summary by alphaXiv.