Reproduce “Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equations” on OpenResearch

About this paper

Liqun Zhao, Keyan Miao, Konstantinos Gatsis, Antonis Papachristodoulou

Researchers from the University of Oxford and the University of Southampton developed a reinforcement learning framework that leverages Neural Ordinary Differential Equations (NODEs) to learn system dynamics, integrating Control Barrier Functions (CBFs) for safety and Control Lyapunov Functions (CLFs) for stability. This system demonstrated superior cumulative rewards and nearly zero safety violations in human-aligned tasks, exhibiting enhanced sample efficiency without relying on predefined physics models.

Key insights

• NODEs can effectively approximate complex system dynamics from data, allowing control-theoretic safety and stability guarantees to be applied without explicit nominal models.
• Integrating formal safety (CBF) and stability (CLF) constraints directly into an actor-critic RL objective via an Augmented Lagrangian method enables robust, human-aligned policy learning.
• A dual-controller architecture, comprising a performance-oriented primary controller and a safety-focused backup controller, provides robust behavior by prioritizing safety when needed.

Summary by alphaXiv.