Reproduce “Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, Sergey Levine
UCBVI-Shaped scales exploration bonuses and caps value estimates using a value hint. Under a pointwise value-bound condition, its regret bound depends on an effective state space and improves a horizon-dependent term. In deterministic maze simulations, it has lower cumulative regret than unshaped UCBVI, especially in a double corridor with many irrelevant paths; these results do not establish gains on general tasks.
Key insights
- Reward shaping can provably improve sample efficiency in RL by enabling the effective pruning of irrelevant or suboptimal regions of the state space from exploration.
- The theoretical regret bound for learning is shown to depend on a reduced 'effective state space' (`|S \ PathPseudoSub_Delta|`), which can be significantly smaller than the total state space.
- Adaptive bonus scaling and value projection within the UCBVI framework not only preserve optimism but also reduce the horizon dependency in regret bounds from `H^2` to `H * beta * ~V_max`, enhancing efficiency in long-horizon tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.