Reproduce “The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lianmin Chen, Junbin Qiu, Chenxing Wei, Yao Shu, Kun He
ZO-BOLT applies zeroth-order optimization to post-training and tests how paired data affects its update estimates. Reusing a minibatch for both perturbed evaluations cancels leading sampling fluctuations, unlike independent sampling, whose oracle-noise bound grows as 1/μ². On GSM8K, matched ZO-BOLT reached 63.78% best accuracy versus 49.35% mismatched, but this does not remove random-direction variability or establish true on-policy performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.