Reproduce “Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Henrik Müller, Daniel Kudenko
This paper presents a method for improving potential-based reward shaping effectiveness by introducing a constant bias to the potential function. This bias aligns the shaped rewards with the agent's initial Q-values and external rewards, leading to faster learning and enhanced sample efficiency in both tabular and deep reinforcement learning environments.
Key insights
- The effectiveness of potential-based reward shaping critically depends on the interaction between the shaped reward, the original external reward, and the agent's initial Q-value estimates.
- A simple constant bias applied to the potential function can effectively tune the shaping signal, making it more exploitable by the learning agent without directly manipulating initial Q-values.
- The scale of potential functions in goal-directed tasks is inherently limited by external rewards and initial Q-values, and continuous potential functions can struggle to consistently provide correct incentives for small changes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.