Reproduce “Multiplication-Free Transformer Training via Piecewise Affine Operations” on OpenResearch

About this paper

Atli Kosson, Martin Jaggi

Researchers at EPFL demonstrated the first end-to-end multiplication-free training of modern neural network architectures, including Transformers, by replacing all arithmetic operations with Piecewise Affine Operations. This approach achieved performance comparable to standard training, with DeiT-Tiny on ImageNet-1k matching baseline accuracy and a small Transformer on IWSLT14 exhibiting only a 0.9 BLEU point reduction.

Key insights

• Neural networks entirely composed of Piecewise Affine Operations are fully and jointly piecewise affine in both their input and weights, offering a new theoretical framework for analysis.
• Approximate derivatives, which mimic the original function's derivative using PAOs, generally perform better during backpropagation than the exact derivatives of the piecewise affine functions themselves.
• The algorithmic viability of comprehensive multiplication-free training provides crucial evidence to motivate the development of specialized hardware that can achieve substantial energy and area savings over traditional processors.

Summary by alphaXiv.