Reproduce “Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wonjoon Goo, Scott Niekum

The Action-Restricted Q-learning (ARQ) algorithm integrates explicit behavior cloning using score-based generative models into a value-based offline reinforcement learning framework. This approach achieves state-of-the-art performance across various D4RL and Robomimic tasks by effectively mitigating distribution shift through a principled action restriction mechanism.

Key insights

  • Explicitly modeling complex, multi-modal behavior policies using high-fidelity generative models is both feasible and advantageous in offline RL.
  • The Direct Q-Penalization framework provides a unified theoretical lens for understanding and designing offline RL algorithms by relating value penalties to policy regularization.
  • For suboptimal datasets, value learning remains essential for surpassing the performance of the behavior policy, even when enhanced by explicit behavior cloning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.