Reproduce “Backpropagation through the Void: Optimizing control variates for black-box gradient estimation” on OpenResearch

About this paper

Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, David Duvenaud

Researchers from the University of Toronto and Vector Institute developed LAX and RELAX, a general framework for estimating unbiased, low-variance gradients for black-box functions of continuous or discrete random variables. The method uses learnable, neural network-parameterized control variates, demonstrating improved training performance, faster convergence, and significant reduction in gradient variance across toy problems, discrete variational autoencoders, and reinforcement learning tasks.

Key insights

• Directly optimizing a neural network-parameterized control variate for gradient variance reduction can yield significantly lower variance for black-box or non-differentiable objectives.
• The framework unifies and generalizes existing variance reduction techniques, making them applicable to a broader range of problems, including discrete variables without requiring specific continuous relaxations for the objective.
• Action-dependent baselines, learned through this framework, offer a more powerful method for reducing policy gradient variance in reinforcement learning compared to standard state-dependent baselines.

Summary by alphaXiv.