Reproduce “Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training” on OpenResearch

About this paper

Hassan Hamad, Yuou Qiu, Peter A. Beerel, Keith M. Chugg

Researchers at USC developed Quantization Aware Approximate Logarithmic Number System (QAA-LNS) arithmetic, optimizing approximations for specific bitwidths to facilitate efficient deep learning training. This approach enables near 32-bit floating-point accuracy with 12-14 bits and yields up to 53.5% energy and 32.5% area savings for Multiply-Accumulate units compared to linear fixed-point designs.

Key insights

• The optimal approximation for LNS arithmetic is critically dependent on the specific bitwidth configuration and must be 'quantization-aware' to maintain numerical stability and accuracy at low precisions.
• Designing piece-wise linear approximations with power-of-two slopes allows for the efficient hardware implementation of complex LNS addition through simple bit-shift operations, rather than costly multiplications.
• An optimization loss function that directly accounts for the effects of quantization (minimizing error between quantized ideal and approximate sums) is essential for stable and accurate low-bitwidth deep learning training.

Summary by alphaXiv.