Reproduce “NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation” on OpenResearch

About this paper

Qinyu Li, Yee Whye Teh, Razvan Pascanu

NoProp trains image classifiers by having each block predict a clean class label from the image and a noisy label, using backpropagation only within that block. At inference, blocks run sequentially to refine noise into a prediction. On CIFAR-10 with one-hot labels, NoProp-DT reached 79.25±0.28% test accuracy versus 79.92±0.14% for backpropagation, while using 0.64 versus 1.17 GB of GPU memory; evidence is limited to small image-classification experiments.

Key insights

• Effective deep learning can be achieved through local, block-wise learning strategies without relying on full end-to-end forward or backward passes.
• Denoising score matching, a technique from generative diffusion models, provides a viable and efficient mechanism for credit assignment in deep neural networks.
• By defining intermediate representations as noisy versions of target label embeddings, NoProp challenges the traditional understanding that globally learned hierarchical representations are strictly necessary for competitive deep learning performance.

Summary by alphaXiv.