Reproduce “Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingchuan Ma, Dongqi Cai, Daqian Yang, Zhenyan Lu, Chen Chen, Teresa Lv, Yan Gao, Shangguang Wang and 3 more
ZO-Curv uses forward-only zeroth-order probes on idle serving devices to estimate damped diagonal curvature, then scales and clips the exact first-order gradient computed by the training node. In the GPT-2 770M comparison, it reaches AdamW’s target validation loss in 3.98 hours, versus 7.53 hours; this training-node timing excludes concurrent query time, with separate disjoint-GPU tests supporting the speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.