Reproduce “Value Explicit Pretraining for Learning Transferable Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kiran Lekkala, Henghui Bao, Sumedh A. Sontakke, Erdem Biyik, Laurent Itti

Value Explicit Pretraining (VEP) introduces a method for learning generalizable visual representations for reinforcement learning by leveraging Monte Carlo value estimates from suboptimal, unlabeled demonstration data within a contrastive pretraining framework. The approach achieved up to a 2-fold improvement in cumulative rewards and a 3-fold improvement in sample efficiency on new tasks across Atari, visual navigation, and continuous control benchmarks.

Key insights

  • Control-specific information, captured by Monte Carlo value estimates, can be effectively leveraged from suboptimal, unlabeled demonstration data to guide representation learning.
  • Using value estimates as a sampling mechanism for contrastive learning (triplet loss) is more effective for learning transferable representations than direct value regression.
  • Representations that explicitly encode a "propensity for success" and are invariant to superficial visual changes lead to significantly improved generalization across diverse visual reinforcement learning tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.