Reproduce “Reinforcement Learning with Videos: Combining Offline Observations with Interaction” on OpenResearch

About this paper

Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, Chelsea Finn

The Reinforcement Learning with Videos (RLV) framework combines large datasets of offline observational videos, including human activities, with online robot interaction to train robot policies. This approach enables robots to learn challenging vision-based skills with a substantial reduction in required interaction samples, achieving comparable performance to standard RL with 30-50% fewer data points while managing significant domain shift.

Key insights

• Effective utilization of action-free, reward-free observational data is possible by estimating actions via an inverse model and assigning heuristic rewards, even if sub-optimal.
• Adversarial domain adaptation is crucial for aligning diverse visual inputs (e.g., human videos and robot camera feeds) into a shared, task-relevant feature space, overcoming significant embodiment and viewpoint differences.
• Integrating processed observational data into an off-policy RL algorithm allows the agent to both benefit from observed behaviors and improve upon them through self-supervision and online interaction.

Summary by alphaXiv.