Reproduce “Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vin Bhaskara, Haicheng Wang
Curiosity-Critic introduces an intrinsic reward for world model training that evaluates cumulative prediction error improvement, allowing it to distinguish between learnable and unlearnable uncertainties. This method achieved a mean L2 prediction error of 1.858 1 0.080 on deterministic cells in a stochastic grid world, significantly outperforming baselines like Curiosity V1 (7.114 1 0.147) by directing exploration towards learnable areas.
Key insights
- Rewarding the cumulative reduction in world model error provides an inherent mechanism to differentiate learnable from unlearnable aspects of an environment.
- A global, cumulative objective for world model improvement can be simplified into a practical, per-step reward signal by considering its asymptotic behavior.
- A learned critic can effectively estimate the irreducible noise floor of a transition, converging faster than the world model itself, thereby providing efficient, targeted exploration guidance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.