Reproduce “Risk-Constrained Reinforcement Learning with Percentile Risk Criteria” on OpenResearch

About this paper

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, Marco Pavone

This paper develops efficient reinforcement learning algorithms for Markov Decision Processes subject to percentile risk constraints, specifically Conditional Value-at-Risk (CVaR) and chance constraints. It demonstrates their ability to learn policies that effectively manage worst-case outcomes in problems like optimal stopping and ad-recommendation by balancing expected performance with explicit risk thresholds.

Key insights

• Percentile risk constraints (CVaR, chance constraints) can be effectively managed in MDPs by transforming them into a Lagrangian saddle-point optimization problem.
• Augmenting the MDP state with the cumulative constraint cost allows standard reinforcement learning algorithms to incorporate complex, path-dependent risk constraints.
• Multi-timescale stochastic approximation provides a robust framework for proving convergence of actor-critic methods in the presence of multiple interacting parameters, such as those in risk-constrained settings.

Summary by alphaXiv.