Reproduce “Contraction-Gauge Preconditioning for Quantized Matrix Multiplication” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Piyush Sao, Narasinga Miniskar, Pedro Valero-Lara, Keita Teranishi, Sudip Seal

Oak Ridge National Laboratory researchers developed Contraction-Gauge Preconditioning, a unified framework for optimizing low-precision quantized matrix multiplication (C=AB). This framework utilizes an exact product-error identity and geometric programming to find optimal diagonal preconditioning, achieving up to 20.5% product error reduction and a 26.4% reduction in logit mean squared error on a three-block image classifier.

Key insights

  • Quantization error in matrix products is fundamentally product-weighted; errors in one factor's entries are scaled by the energy of the opposite factor's corresponding rows or columns.
  • Optimal diagonal scaling for grouped factors can be cast as a convex geometric program, offering a certifiably optimal solution within this transform family.
  • Computable decision statistics, such as log-magnitude clustering for partitioning and slice-energy anti-correlation for hierarchical rotations, systematically guide transform selection beyond heuristics.
  • Stochastic error models derived for preconditioning demonstrate high fidelity in ranking candidates for practical deterministic round-to-nearest (RTN) quantization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.