Reproduce “One weird trick for parallelizing convolutional neural networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alex Krizhevsky
This work introduces a hybrid parallelization strategy for convolutional neural networks, leveraging data parallelism for convolutional layers and model parallelism for fully-connected layers. The method achieved a 6.16x speedup on 8 GPUs for ImageNet training while maintaining high accuracy, partly due to a variable batch size technique.
Key insights
- Optimizing parallelization requires recognizing the distinct computational and memory demands of different CNN layers, where convolutional layers benefit from data parallelism and fully-connected layers from model parallelism.
- Maintaining a smaller effective batch size for fully-connected layers, even when convolutional layers process significantly larger batches, helps preserve model accuracy and improves final solution quality.
- Sub-linear scaling in multi-GPU training is primarily driven by inefficient dense matrix multiplications in upper layers, the characteristics of one-to-all communication schemes, and hardware-specific PCI-Express topology limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.