Reproduce “Reward-Weighted Regression Converges to a Global Optimum” on OpenResearch

About this paper

Miroslav Štrupl, Francesco Faccio, Dylan R. Ashley, Rupesh Kumar Srivastava, Jürgen Schmidhuber

Researchers at The Swiss AI Lab IDSIA formally proved that Reward-Weighted Regression (RWR) converges to a global optimum in general compact state and action spaces, under conditions of no function approximation and only linear reward transformations. They also demonstrated that commonly used non-linear reward transformations can inadvertently alter an MDP's optimal policy, highlighting a critical consideration for EM-based reinforcement learning.

Key insights

• The RWR update rule $\pi^{n+1}(a|s) = \frac{Q_{\pi^n}(s,a)\pi^n(a|s)}{V_{\pi^n}(s)}$ guarantees monotonic improvement in value functions, ensuring non-decreasing performance with each iteration.
• Common non-linear reward transformations (e.g., exponential scaling) can fundamentally alter an MDP's optimal policy, underscoring the need for careful reward function design and analysis.
• A specialized concept of weak convergence for policy measures is crucial for proving global optimality in continuous domains where optimal policies might be deterministic or non-unique for a given state.

Summary by alphaXiv.