Reproduce “Offline Reinforcement Learning with Behavioral Supervisor Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Padmanaba Srinivasan, William Knottenbelt

Researchers at Imperial College London developed TD3 with Behavioral Supervisor Tuning (TD3-BST), an offline reinforcement learning algorithm that utilizes Morse neural networks to dynamically regularize policy learning based on state-action uncertainty. This method achieved competitive and often leading performance across D4RL Gym Locomotion and Antmaze benchmarks, critically, without requiring per-dataset hyperparameter tuning.

Key insights

  • Morse neural networks can effectively quantify epistemic uncertainty for state-action pairs in offline RL, robustly distinguishing between in-dataset and out-of-distribution samples.
  • Dynamic regularization, informed by this uncertainty, allows for competitive policy learning without the need for extensive per-dataset hyperparameter tuning, addressing a major practical limitation.
  • The proposed Behavioral Supervisor Tuning (BST) objective is modular and can improve performance even when integrated into one-step methods like IQL, suggesting broad applicability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.