Reproduce “Reinforcement Learning with Perturbed Rewards” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jingkang Wang, Yang Liu, Bo Li

A robust reinforcement learning framework is introduced to enable agents to learn optimal policies from biased, noisy reward signals, which are common in real-world applications. The framework uses an unbiased surrogate reward derived from a dynamically estimated confusion matrix, allowing various deep reinforcement learning algorithms to achieve consistently higher expected returns and faster convergence compared to learning directly with noisy rewards.

Key insights

  • Q-Learning, when updated with the proposed unbiased surrogate rewards, is proven to converge almost surely to the optimal Q-function, providing theoretical guarantees for the approach.
  • Learning with surrogate rewards introduces increased variance, with sample complexity for achieving a near-optimal policy scaling by `O(1/det(C)^2)`, highlighting the 'price' of learning in highly noisy environments.
  • The framework demonstrates that insights from robust supervised learning with noisy labels (e.g., unbiased estimators) can be effectively adapted to address challenging biased reward noise in reinforcement learning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.