Reproduce “Information Directed Reward Learning for Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Lindner, Matteo Turchetta, Sebastian Tschiatschek, Kamil Ciosek, Andreas Krause
The paper introduces Information Directed Reward Learning (IDRL), a method that efficiently learns reward functions for Reinforcement Learning by strategically selecting expert queries. It achieves improved sample efficiency across various environments, including Deep RL settings, by prioritizing information that distinguishes between plausibly optimal policies.
Key insights
- Directing active reward learning to reduce uncertainty about differences in returns between plausibly optimal policies, rather than uniformly improving reward model accuracy, leads to significantly higher sample efficiency.
- An information-theoretic framework, leveraging entropy and mutual information, can provide a unified and general approach for selecting optimal queries across diverse expert feedback types.
- The trade-off between computational cost and query efficiency can be managed, allowing IDRL to scale to complex Deep RL environments while still outperforming baselines in terms of required expert feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.