Reproduce “PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, Kenji Doya
PIPPS uses particle rollouts and total propagation (TP) to support model-based policy search when gradients through long nonlinear trajectories become unreliable. TP combines likelihood-ratio and reparameterization gradients, weighting them by estimated variance at each backward step. In cart-pole swing-up with Angle Cost and noise multiplier k = 1, TP reached 0.96 final-trial success versus PILCO’s 0.79, in the reported setting.
Key insights
- Reparameterization (RP) gradients suffer from "exploding gradient variance" in long chains of nonlinear computations, a phenomenon termed "the curse of chaos," where gradient directions become arbitrary despite a smooth objective.
- Likelihood Ratio (LR) gradients are robust to the "curse of chaos" and yield smoother gradient estimates, though they typically have higher variance than RP gradients in stable regions.
- An adaptive combination of RP and LR gradients via the Total Propagation algorithm can achieve significantly lower gradient variance than either method alone, leveraging RP's efficiency where stable and LR's robustness in chaotic regions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.