Reproduce “Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, Sergey Levine

Leave No Trace introduces a reinforcement-learning framework in which a learned reset policy both restores the environment and serves as a recoverability model for screening risky actions. Across gridworld and simulated robotic tasks, early intervention and repeated recovery reduced manual hard resets—by up to an order of magnitude in cliff-cheetah and 78% in gridworld—while enabling autonomous data collection and an emergent curriculum for peg insertion.

Key insights

  • A learned recovery policy can define a changing safety boundary: as recovery improves, the set of states from which the agent can safely explore expands.
  • Automated resets are useful even in reversible tasks because they repeatedly expose the forward policy to relevant starting conditions instead of letting it exploit already-solved states.
  • The interaction between forward and reset policies generates an implicit curriculum, presenting increasingly difficult recovery configurations without manually specifying them.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.