Reproduce “Imitation Learning via Off-Policy Distribution Matching” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ilya Kostrikov, Ofir Nachum, Jonathan Tompson

ValueDICE introduces an algorithm for entirely off-policy imitation learning by directly optimizing a KL-divergence objective between the learned policy and expert distributions. This method achieves state-of-the-art sample efficiency and faster convergence across continuous control tasks, especially in low-data regimes, without requiring a separate reinforcement learning optimization loop.

Key insights

  • The Donsker-Varadhan representation, combined with a Bellman-like transformation, can yield a truly off-policy objective that accurately minimizes the KL-divergence between policy and expert state-action distributions.
  • The function 'nu' in ValueDICE implicitly learns the future discounted sum of rewards corresponding to the distribution ratio, allowing direct policy optimization from the same objective.
  • Global optimality of matching the expert policy is preserved even with a small regularization term that mixes replay buffer data, enhancing robustness during training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.