Reproduce “How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Augustine N. Mavor-Parker, Kimberly A. Young, Caswell Barry, Lewis D. Griffin
Aleatoric Mapping Agents (AMAs) are introduced to explicitly model and discount irreducible randomness (aleatoric uncertainty) in state transitions, enabling robust exploration in stochastic environments. This approach effectively avoids the noisy TV problem, demonstrating superior performance over existing curiosity-driven methods like RND and ensemble disagreement in various Atari games and MiniGrid environments.
Key insights
- Existing state-of-the-art exploration methods like Random Network Distillation (RND) and ensemble disagreement are vulnerable to action-dependent noisy TVs, especially when the stochasticity introduces a large number of novel states.
- Explicitly distinguishing between epistemic (reducible) and aleatoric (irreducible) uncertainty provides a robust and efficient mechanism for intrinsic motivation that allows agents to 'stay curious' about truly learnable aspects while 'avoiding noisy TVs'.
- The computational framework of AMAs offers a biologically plausible model for how neuromodulators like acetylcholine might signal expected uncertainty in the brain, suggesting testable hypotheses for experimental neuroscience.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.