Reproduce “PEGASUS: A Policy Search Method for Large MDPs and POMDPs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Y. Ng, Michael I. Jordan
PEGASUS (Policy Evaluation-of-Goodness And Search Using Scenarios) from UC Berkeley addresses sequential decision-making under uncertainty by transforming POMDPs into equivalent deterministic systems, enabling efficient direct policy search. This method achieves polynomial sample complexity dependence on the horizon time, outperforming previous exponential bounds, and demonstrated effective control on a bicycle simulator, achieving median riding distances of 0.995-1.07 km.
Key insights
- Even if optimal value functions are complex, simple policies can often achieve near-optimal performance.
- By exposing and fixing randomness via a deterministic simulative model, the stochasticity of the environment can be converted into deterministic state evolution in an expanded state space.
- Reusing fixed random scenarios across policy evaluations reduces variance in policy comparisons while maintaining unbiased estimates, enabling efficient policy optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.