Reproduce “Assessing the Potential for Catastrophic Failure in Dynamic Post-Training Quantization” on OpenResearch

About this paper

Logan Frank, Paul Ardis

Research identifies and quantifies the risk of catastrophic failure in dynamic post-training quantization (DPTQ), where specific network and policy pairings can lead to severe accuracy degradation for a subset of inputs. The study shows that a ResNet18 model can experience a 24.39% accuracy reduction after DPTQ, in contrast to a 0.22% reduction for a robustly trained counterpart with similar full-precision performance.

Key insights

• Catastrophic failure in DPTQ is a demonstrable phenomenon, arising from the specific interplay between the network and its dynamic quantization policy, rather than the network's inherent brittleness alone.
• Different network architectures exhibit unique layer-specific vulnerabilities, with activation distributions characterized by long tails and sparsity being particularly susceptible to quantization errors.
• Pre-deployment analysis of full-precision model robustness to common input corruptions might serve as an initial indicator for potential brittleness to DPTQ-induced catastrophic failures.

Summary by alphaXiv.