Reproduce “Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz and 2 more
Recovery RL presents a safe reinforcement learning algorithm featuring a dual-policy architecture for task and recovery and leveraging offline data for learned recovery zones. It achieves a 2 to 20 times more efficient trade-off between task successes and constraint violations in simulations and 3 times greater efficiency on a physical robot, outperforming prior methods.
Key insights
- Decoupling task and safety objectives into distinct policies (task and recovery) avoids inherent conflicts in joint optimization, leading to more efficient learning and safer exploration.
- Pre-training safety mechanisms with offline data significantly reduces the necessity for costly online constraint violations, making real-world RL deployment more practical.
- Action relabeling for the task policy allows it to learn effectively as if operating in a 'safer' environment, improving its ability to maximize reward under safety constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.