Reproduce “Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yihan Du, Siwei Wang, Longbo Huang

This research introduces Iterated Conditional Value-at-Risk (CVaR) Reinforcement Learning and Worst Path Reinforcement Learning, offering frameworks for continuous risk-sensitive decision-making in unknown environments. It provides the first provably efficient algorithms (ICVaR-RM, ICVaR-BPI, MaxWP) with finite-time regret and sample complexity guarantees for these formulations, demonstrating performance similar to risk-neutral RL when risk tolerance is high, and constant regret for extreme risk aversion.

Key insights

  • Iterated CVaR allows for continuous risk control by focusing on the worst \alpha-portion of future rewards at each decision step, crucial for safety-critical domains.
  • Novel analytical techniques, including a method to bound CVaR changes and the use of a distorted visitation distribution, were developed to handle the non-linearity of the CVaR operator and its impact on exploration.
  • Worst Path RL (as \alpha approaches 0) exhibits distinct properties, achieving constant regret once environmental connectivity is sufficiently learned, which provides unique insights for extreme risk aversion.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.