Reproduce “Structured Transforms for Small-Footprint Deep Learning” on OpenResearch

About this paper

Vikas Sindhwani, Tara N. Sainath, Sanjiv Kumar

Researchers at Google introduced structured transforms based on low displacement rank theory, enabling deep neural networks that are significantly more compact and computationally efficient. This framework provides substantial speedups (10-100x) for inference and training while maintaining competitive accuracy across image classification and achieving near state-of-the-art performance in mobile keyword spotting with 3-80x fewer parameters than larger models.

Key insights

• The classical theory of low displacement rank provides a principled, theoretically grounded approach to design deep learning transforms that are simultaneously compact and computationally efficient.
• The proposed structured transforms, such as Toeplitz-like matrices, generalize 1D convolutional layers and offer a tunable continuum of modeling capacity via the displacement rank hyperparameter.
• Leveraging FFTs for f-Circulant matrices enables dramatic acceleration of core deep learning operations, transforming dense matrix multiplications into O(N log N) complexity, crucial for both inference and training.

Summary by alphaXiv.