Reproduce “For SALE: State-Action Representation Learning for Deep Reinforcement Learning” on OpenResearch

About this paper

Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Shane Gu, Doina Precup, David Meger

Researchers from Mila, McGill University, and Google DeepMind developed State-Action Learned Embeddings (SALE) and Policy Checkpoints to improve deep reinforcement learning in continuous control tasks with low-level state observations. Their integrated algorithm, TD7, achieved a 276.7% average performance gain over TD3 at 300k timesteps on MuJoCo benchmarks by enhancing sample efficiency and stability.

Key insights

• Even compact, low-level state observations benefit significantly from learned state-action representations that capture underlying dynamical system complexities, expanding the applicability of representation learning in RL.
• Expanding action-dependent inputs through representation learning can unexpectedly induce extrapolation error in *online* RL, which can be effectively and simply mitigated by clipping target Q-values.
• Decoupled training of representation encoders (as an auxiliary loss) from the main RL objective, combined with appropriate normalization (like AvgL1Norm), is crucial for stable and effective representation learning in continuous control.

Summary by alphaXiv.