Reproduce “Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning” on OpenResearch

About this paper

Eugene Belilovsky, Louis Leconte, Lucas Caccia, Michael Eickenberg, Edouard Oyallon

Decoupled Greedy Learning (DGL) enables training Convolutional Neural Networks by optimizing each layer independently using a local objective, thereby resolving backpropagation's "update locking" and "forward locking" inefficiencies. DGL achieves comparable or superior accuracy to standard backpropagation on ImageNet (e.g., matching ResNet-152's 74.4% Top-1 accuracy) while offering substantial speedups and enabling asynchronous distributed training with up to 28x memory and 23x bandwidth compression via online quantization.

Key insights

• Decoupling network training through a greedy, layerwise objective can achieve performance comparable to or better than standard backpropagation while allowing for high degrees of parallelism across modules.
• The computational efficiency of DGL is significantly enhanced by using lightweight auxiliary networks (e.g., MLP-SR-aux) that provide local loss signals with minimal overhead.
• Online vector quantization effectively mitigates bandwidth and memory bottlenecks in asynchronous distributed learning, enabling substantial compression (e.g., >15x) with minimal impact on model accuracy.

Summary by alphaXiv.