Reproduce “DeepSpark: A Spark-Based Distributed Deep Learning Framework for Commodity Clusters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanjoo Kim, Jaehong Park, Jaehee Jang, Sungroh Yoon
DeepSpark, a distributed deep learning framework built on Apache Spark, accelerates deep neural network training on commodity clusters by employing a lock-free asynchronous Elastic Averaging SGD and adaptive communication. The framework achieved a 6.0x speed-up at 0.6 accuracy against CaffeOnSpark and a 2.5x speed-up at 0.7 accuracy against single-node Caffe, also reaching 9% higher test accuracy.
Key insights
- A lock-free asynchronous EASGD strategy, coupled with an adaptive communication period, effectively reduces communication overhead and achieves faster wall-clock convergence on commodity hardware.
- While asynchronous updates introduce a discrepancy penalty requiring more iterations for convergence, the gains from reduced communication time significantly outweigh this, leading to overall speed-up.
- The derived speed-up model shows that synchronous distributed training struggles on commodity clusters where communication overhead is high, highlighting the benefit of DeepSpark's asynchronous approach.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.