Reproduce “Safe Reinforcement Learning using Ideas from Model Predictive Control” on OpenResearch

About this paper

Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender

A research framework integrates Deep Reinforcement Learning (DRL) with Model Predictive Control (MPC) to ensure strict safety in cyber-physical systems. It establishes a recursively feasible state-action space offline using MPC and employs an online projection filter to map DRL-proposed actions into this verified safe space, achieving stable policy convergence on a physical testbed with significantly reduced constraint violations during training.

Key insights

• Proactive safety guarantees can be achieved by offloading complex predictive calculations to an offline phase, enabling real-time, lightweight safety enforcement during online learning and deployment.
• For control-affine systems, the feasible action space often forms a convex set, allowing for efficient mapping of safety boundaries through bisection search rather than exhaustive sampling.
• The concept of a 'point of no return' is explicitly captured by the shrinking of the feasible action space as a system approaches physical limits at high velocities, necessitating forward-looking safety measures.

Summary by alphaXiv.