Reproduce “Deep SPI: Safe Policy Improvement via World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Florent Delgrange, Raphael Avalos, Willem Röpke
This research introduces DEEP SPI, a framework and algorithm for safe policy improvement in online deep reinforcement learning environments that utilize world models and representation learning. It provides theoretical guarantees for monotonic policy improvement and convergence, while empirically demonstrating competitive performance on the ALE-57 benchmark.
Key insights
- A novel neighborhood operator for constraining policy updates based on extremal importance ratios guarantees monotonic policy improvement and convergence to the optimal value function in finite state-action spaces.
- Local reward and transition losses can effectively bound the discrepancy between a policy's performance in the true environment and a learned world model, addressing the out-of-trajectory problem by ensuring model reliability within defined policy deviation limits.
- Minimizing these local losses implicitly enforces representation quality, ensuring that the value function maintains 'almost' Lipschitz continuity in the latent space, thereby mitigating confounding policy updates by preserving distinctiveness for states with similar values.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.