Reproduce “Risk-Sensitive Exponential Actor Critic” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alonso Granados, Jason Pacheco
A new model-free deep reinforcement learning algorithm, Risk-Sensitive Exponential Actor-Critic (rsEAC), integrates the entropic risk measure by employing novel techniques to stabilize exponential value function learning and policy gradient updates. The approach enables reliable acquisition of risk-sensitive policies across various continuous control tasks, demonstrating effective risk modulation and improved performance over existing baselines.
Key insights
- Policy gradient theorems for the entropic risk measure can be derived to avoid high-variance full-trajectory estimates, providing a more stable theoretical foundation for risk-sensitive optimization.
- Directly learning exponential value functions causes severe numerical instabilities; reparameterizing them as e^Q(s,a) and performing gradient computations in the log-domain significantly enhances stability.
- A combination of log-domain normalization and exponent clipping is essential to prevent numerical overflow/underflow and manage exploding/vanishing gradients when using deep function approximators for entropic risk.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.