Reproduce “SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Siddharth Reddy, Anca D. Dragan, Sergey Levine

Soft Q Imitation Learning (SQIL) introduces a straightforward imitation learning approach that modifies off-policy Q-learning with a sparse reward scheme and balanced replay buffer sampling. This method enables agents to overcome distribution shift and effectively learn from demonstrations in various tasks, outperforming Behavioral Cloning and demonstrating competitive or superior performance against Generative Adversarial Imitation Learning, particularly in image-based environments.

Key insights

  • A simple, sparse reward structure (+1 for expert, 0 for agent) combined with off-policy reinforcement learning can provide the necessary incentives for long-horizon imitation and recovery from out-of-distribution states.
  • SQIL is theoretically equivalent to a regularized variant of Behavioral Cloning, where the regularization implicitly incorporates a sparsity prior and leverages environmental dynamics.
  • The Bellman equation in Q-learning effectively propagates value backward, enabling the agent to learn actions that lead to demonstrated states, even if those actions aren't directly demonstrated.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.