Reproduce “The Optimal Reward Baseline for Gradient-Based Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lex Weaver, Nigel Tao
The authors introduce GARB, subtracting an average-reward baseline before weighting an eligibility trace in policy-gradient estimation. This preserves the gradient’s expectation, while theory says the variance-minimizing constant approaches average reward as discounting approaches one. A three-state test found a shift from 0.6 times average reward at 0.4 to near average reward at 0.99, but finite-discount optimality is not universal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.