Reproduce “Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Paul Vicol, Zico Kolter, Kevin Swersky

The ES-Single algorithm introduces an unbiased and constant-variance gradient estimator for unrolled computation graphs, addressing limitations in prior Evolution Strategies-based methods. This approach enhances optimization stability and efficiency for long-horizon machine learning tasks requiring frequent truncated unrolls.

Key insights

  • ES-Single exhibits a constant variance profile with respect to the number of partial unrolls, contrasting with PES where variance increases linearly.
  • The method provides unbiased gradient estimates by design due to its equivalence to full-unroll ES, particularly for quadratic objectives.
  • The simplified perturbation sampling mechanism makes ES-Single easier to implement and computationally more efficient per iteration compared to PES.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.