Reproduce “Model-Value Inconsistency as a Signal for Epistemic Uncertainty” on OpenResearch

About this paper

Angelos Filos, Eszter Vértes, Zita Marinho, Gregory Farquhar, Diana Borsa, Abram Friesen, Feryal Behbahani, Tom Schaul and 2 more

The authors introduce implicit value ensembles (IVEs) that estimate uncertainty by applying one model for different rollout lengths and evaluating remaining returns with one value function. Disagreement among these estimates, called self-inconsistency, serves as a proxy for epistemic uncertainty without extra ensemble members. In a withheld-state gridworld, disagreement was higher around the unvisited state; this does not establish calibrated uncertainty.

Key insights

• Model-value inconsistency arises naturally in model-based RL agents due to the different compositions and training targets of multi-step value predictions, providing diversity similar to an explicit ensemble.
• Disagreement between short-term and long-term model-predicted values can signal areas where the agent's learned model or value function is unreliable, indicating epistemic uncertainty.
• The IVE provides both an averaged value prediction ($\mu$-IVE) for robust planning and an uncertainty estimate ($\sigma$-IVE) for guiding exploration or ensuring robustness.

Summary by alphaXiv.