Reproduce “Guided Policy Optimization under Partial Observability” on OpenResearch

About this paper

Yueheng Li, Guangming Xie, Zongqing Lu

The Guided Policy Optimization (GPO) framework enables efficient utilization of privileged information in partially observable reinforcement learning by co-training a guider with full state access and a learner with partial observations, ensuring the guider's policy remains imitable. This approach consistently achieved superior performance across didactic, noisy continuous control, and memory-based tasks compared to existing methods.

Key insights

• The learner's policy updates can be viewed as a constrained policy mirror descent, providing theoretical guarantees for policy improvement towards optimality.
• The framework effectively divides the learning challenge, with the guider handling complex RL gradient computations using privileged information and the learner performing a simpler supervised task.
• Explicitly constraining the guider's policy (backtracking) is critical to prevent the "imitation gap" and ensure continuous, actionable supervision for the learner.

Summary by alphaXiv.