Reproduce “GIPO: Gaussian Importance Sampling Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Baigui Sun, Yang Liu

GIPO adjusts how replay-heavy policy learning uses older experience by smoothly weighting policy-gradient contributions according to the log difference between current and behavior-policy action probabilities. On 10 Meta-World tasks with five seeds per configuration in the stale-replay regime, GIPO with σ=1.0 achieves an Interquartile Mean normalized score of 0.730, versus 0.412 for SAPO and 0.180 for PPO; this result is limited to the tested regime.

Key insights

  • Hard clipping in PPO causes "utilization collapse" by discarding valuable information from stale off-policy data; a smooth, continuously differentiable weighting can leverage this data more effectively.
  • Applying a Gaussian kernel in the log-ratio space provides symmetric trust and boundedness, preventing gradient explosion for high ratios and suppressing irrelevant samples from vanishing ratios.
  • The tunable scale parameter in the Gaussian weight offers a principled way to control the bias-variance trade-off, enabling adaptability to different degrees of policy lag and improving stability under finite-sample estimation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.