Reproduce “Learning Reward Functions by Integrating Human Demonstrations and Preferences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, Dorsa Sadigh
Researchers at Stanford University developed DemPref, a framework that integrates human demonstrations and preference queries to efficiently learn personalized reward functions for autonomous robots. This approach significantly accelerates reward learning convergence in simulation and achieves higher user-rated task success and preference for robot training compared to pure Inverse Reinforcement Learning, particularly benefiting from a novel Iterated Correction mechanism.
Key insights
- Demonstrations and preferences offer complementary strengths: demonstrations provide a coarse initial understanding, while preferences refine human intent more accurately.
- Initializing with even a single demonstration significantly reduces the search space for preference learning, accelerating convergence compared to starting from scratch.
- Using ranking-based feedback with a Plackett-Luce model extracts more information per query than binary preferences, and the Iterated Correction mechanism is especially effective in improving convergence from low-quality initial demonstrations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.