Reproduce “Position: Good Embodied Reward Models Need Bad Behavior Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ran Tian, Yilin Wu, Andrea Bajcsy
Embodied reward models systematically over-reward suboptimal robot behaviors, especially as task complexity increases, due to a lack of negative data in training. Incorporating real robot failure videos with dense reward labels significantly improves model calibration and alignment with human preferences, yielding up to a 10% accuracy increase in distinguishing good from bad robot actions.
Key insights
- State-of-the-art embodied reward models exhibit systematic over-optimism, over-rewarding behaviors that humans perceive as unsatisfactory or unsafe.
- Text-only descriptions of failures are insufficient for improving VLM-based reward models, as they struggle to ground abstract criteria in concrete physical reality.
- Explicit, visually grounded negative feedback, particularly dense, per-timestep reward labels associated with failure examples, is crucial for effectively calibrating embodied reward models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.