Reproduce “Video Prediction Models as Rewards for Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner and 1 more

This research introduces VIPER (Video Prediction Rewards), a method for generating action-free reward signals for reinforcement learning agents by utilizing pre-trained video prediction models. The approach demonstrates that log-likelihoods from autoregressive video models can guide RL agents to achieve expert-level performance across diverse robotic and game environments, including successful generalization to novel robot-task combinations.

Key insights

  • Pre-trained autoregressive video prediction models can provide dense, temporally consistent reward signals for reinforcement learning by leveraging their ability to model sequential data distributions.
  • Using log-likelihoods as rewards, combined with an exploration bonus, effectively aligns the agent's behavior with the expert's distribution while mitigating mode collapse often seen in adversarial imitation learning.
  • Appropriate video model architecture and training strategies, such as masking irrelevant information (Atari) or frame subsampling (RLBench), are crucial for learning effective reward functions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.