Reproduce “Subtask-Aware Visual Reward Learning from Segmented Demonstrations” on OpenResearch

About this paper

Changyeon Kim, Minho Heo, Doohyun Lee, Jinwoo Shin, Honglak Lee, Joseph J. Lim, Kimin Lee

The REDS framework addresses the challenge of reward engineering in reinforcement learning by learning reward functions directly from action-free video demonstrations, integrating subtask segmentation to provide context-aware reward signals. This approach enables robotic agents to learn and complete complex, long-horizon manipulation tasks, outperforming prior visual reward learning methods and sometimes even human-engineered rewards on benchmarks like Meta-World and real-world FurnitureBench tasks.

Key insights

• Complex, long-horizon tasks can be decomposed into simpler subtasks, and learning rewards contextually for each subtask improves performance.
• A multi-component training objective combining EPIC loss for reward alignment, a regularization term for progressive rewards, and contrastive learning for subtask inference enables robust reward function learning.
• Action-free video demonstrations, when segmented into subtasks, provide sufficient information for learning effective reward functions for robotic manipulation.

Summary by alphaXiv.