Reproduce “Training and Inference with Integers in Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuang Wu, Guoqi Li, Feng Chen, Luping Shi
WAGE enables end-to-end training and inference of Deep Neural Networks using only low-bitwidth integer arithmetic, making DNNs suitable for energy-efficient on-device learning. The framework achieves strong performance on datasets like MNIST and CIFAR10, demonstrating that full integer dataflow is feasible for both forward and backward passes.
Key insights
- Error signals primarily guide convergence through their relative values or "orientations" rather than absolute orders of magnitude, allowing for aggressive quantization.
- Maintaining a higher bitwidth "buffer space" (e.g., 8 bits) for gradient accumulation during training is critical for stable convergence and competitive accuracy, even when inference uses lower bitwidths.
- The inherent discretization and stochasticity introduced by low-bitwidth backpropagation can act as a form of regularization, potentially improving generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.