Reproduce “Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexandra Volkova, Matin Ansaripour, Erik Schultheis, Christoph H. Lampert, Dan Alistarh
Q-PACE targets low-precision language-model training by assigning mixed precision to layers according to estimated quantization sensitivity, then periodically reallocating formats under a fixed weight-precision budget. It estimates sensitivity from loss changes under weight perturbations. In 500M-parameter pretraining at a 6.5-bit budget, final test loss was 2.663, versus 2.672 for SNIP and 2.683 for NVIDIA; this one-seed result does not measure total training memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.