Reproduce “Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu

Researchers from Southeast University and 2030 Lab introduced Safe-CRL, a framework that corrects the systematic overestimation bias in existing Contrastive Reinforcement Learning methods when applied to failure-terminated environments. The framework enhances goal-reaching capabilities and survival time across various robotic tasks by leveraging only one-bit failure signals, demonstrating improved robustness and maintaining scalability.

Key insights

  • In failure-terminated Markov Decision Processes, the true goal-reaching Q-function is a sub-probability measure, where its total mass (survival mass) is typically less than 1.
  • Established CRL's trajectory-wise normalization of pre-failure futures systematically overestimates the true goal-reaching value by implicitly assuming unit survival mass for all segments.
  • Explicitly incorporating the survival mass into both critic training and actor optimization, using only the intrinsic one-bit failure signal, is crucial for learning safe and robust policies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.