Reproduce “Reinforcement Learning with Feedback from Multiple Humans with Diverse Skills” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taku Yamagata, Ryan McConville, Raul Santos-Rodriguez
Researchers at the University of Bristol developed a method that enables reinforcement learning agents to learn effectively from multiple human trainers with unknown and varying skill levels. The approach extends the Advise algorithm by incorporating an online Expectation-Maximization process to estimate each trainer's consistency, demonstrating improved performance in a Pac-Man grid world compared to fixed consistency assumptions and standard Q-Learning.
Key insights
- Online estimation of trainer consistency is critical for robust performance, as fixing consistency to incorrect values severely degrades system effectiveness.
- Trainers with consistency levels farther from 0.5 (whether consistently right or consistently wrong) prove more helpful than those close to 0.5, as the system can invert consistently wrong feedback.
- The method successfully extends 'learning from crowds' principles to the reinforcement learning domain, enabling effective aggregation of diverse human feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.