Reproduce “CURL: Contrastive Unsupervised Representations for Reinforcement Learning” on OpenResearch

About this paper

Aravind Srinivas, Michael Laskin, Pieter Abbeel

CURL introduces a method that integrates contrastive unsupervised representation learning with off-policy reinforcement learning to significantly boost the sample efficiency of agents operating directly from pixels. This approach enables pixel-based agents to achieve performance levels comparable to those using ground-truth state features across various continuous and discrete control tasks.

Key insights

• Learning temporal features through consistent random cropping across stacked frames in contrastive learning generally outperforms purely visual discrimination, particularly for tasks requiring complex coordination.
• The contrastive objective can effectively learn broadly useful and potentially reward-agnostic representations, as demonstrated by near-optimal policies even when the encoder's gradients are detached from the RL objective.
• A simple design, including random cropping as the primary augmentation and no extra hyperparameters for balancing the losses, can lead to substantial improvements in data efficiency without adding significant complexity.

Summary by alphaXiv.