Reproduce “ADaPTION: Toolbox and Benchmark for Training Convolutional Neural Networks with Reduced Numerical Precision Weights and Activation” on OpenResearch

About this paper

Moritz B. Milde, Daniel Neil, Alessandro Aimar, Tobi Delbruck, Giacomo Indiveri

ADaPTION, a Caffe-based toolbox, enables training convolutional neural networks with reduced numerical precision for both weights and activations using fixed-point notation. This approach maintains high classification accuracy, achieving 67.5% Top-1 accuracy for VGG16 on ImageNet with 16-bit fixed-point, and significantly increases activation sparsity for efficient deployment on resource-constrained hardware.

Key insights

• Quantizing activations to low precision is critical for hardware efficiency but necessitates specific fine-tuning, as one-time rounding without adaptation drastically reduces accuracy.
• Layer-wise bit allocation for fixed-point representation significantly improves accuracy compared to global settings by adapting to the diverse dynamic ranges across network layers.
• Stochastic rounding is crucial during low-precision training to counteract "fixed points" in parameter space, preventing sudden accuracy drops and aiding stable convergence.

Summary by alphaXiv.