Reproduce “Policy-shaped prediction: avoiding distractions in model-based reinforcement learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Miles Hutson, Isaac Kauvar, Nick Haber
Stanford University researchers developed Policy-Shaped Prediction (PSP), an approach for model-based reinforcement learning that improves robustness against visual distractions by using policy gradients to focus world model learning on task-relevant image regions. The method demonstrated a 2x improvement in robustness on challenging distractor environments while maintaining performance on standard control tasks.
Key insights
- Policy gradients effectively identify task-relevant visual information, enabling a world model to focus its learning on crucial image regions.
- Aggregating pixel-level policy gradient weights using object segmentation (e.g., with SAM) improves stability and effectiveness by focusing on coherent, task-relevant objects.
- An adversarial mechanism can prevent the world model from redundantly encoding self-induced action information, freeing up capacity for true environmental states.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.