Reproduce “Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cong Xie, Oluwasanmi Koyejo, Indranil Gupta
Zeno is a distributed Stochastic Gradient Descent algorithm that tolerates an arbitrary number of faulty workers, requiring only at least one non-faulty worker, a significant departure from prior majority-based fault-tolerance methods. It employs a suspicion-based aggregation rule to filter malicious gradients, demonstrating convergence for non-convex problems even with a majority of faulty nodes and achieving superior accuracy compared to baselines in severe fault scenarios.
Key insights
- Effective fault tolerance in distributed SGD can be achieved with only one non-faulty worker, challenging the conventional 'majority non-faulty' assumption.
- A server-side, post-hoc loss evaluation based on a 'stochastic zero-order oracle' can objectively assess gradient quality, preventing faulty workers from manipulating their scores.
- Zeno's aggregation rule ensures theoretical convergence for non-convex problems at a rate matching fault-free SGD, even in the presence of non-i.i.d. data and a majority of faulty workers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.