Reproduce “Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Sergey Levine
Q-Prop combines replay-trained off-policy critics with an on-policy correction. It uses a critic’s local action-value slope as a control variate, reducing noise while retaining a correction from observed returns. On HalfCheetah-v1, conservative Q-Prop reaches average reward 4000 using about ten times fewer samples than TRPO, a result limited to this simulated benchmark.
Key insights
- An off-policy critic can effectively reduce policy gradient variance without introducing bias, even if the critic is not perfectly accurate, provided it is correlated with the true advantage function.
- The framework unifies policy gradient and actor-critic methods, viewing the approach as using an off-policy critic for variance reduction or on-policy Monte Carlo returns for bias correction.
- Conservative adaptive control of the critic's influence (c-Q-Prop) is critical for maintaining robust learning stability, especially when the critic might be unreliable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.