Reproduce “Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients” on OpenResearch

About this paper

Parisa Davar, Frédéric Godin, Jose Garrido

A new policy gradient algorithm, POTPG, integrates Extreme Value Theory (EVT) to robustly minimize catastrophic risk in sequential decision-making, particularly when extreme events are rare. This method provides more accurate and stable estimation of high-confidence Conditional Value-at-Risk (CVaR) than empirical approaches, leading to improved policy optimization in simulated and real-world financial hedging environments.

Key insights

• Extreme Value Theory offers a robust statistical framework for modeling the far tail of distributions, providing more stable and accurate estimates of high-confidence CVaR than empirical methods, even with scarce extreme data.
• The POT approach enables extrapolation of extreme event behavior from large, but not necessarily maximally extreme, observations, effectively overcoming limitations of empirical sample averaging for heavy-tailed cost distributions.
• Using the same dynamically determined threshold for both the current and perturbed policies in finite-difference gradient estimation significantly enhances the stability and reduces noise in policy gradient calculations.

Summary by alphaXiv.