Reproduce “Self-Supervised Exploration via Disagreement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta
Researchers from UC Berkeley, CMU, and Facebook AI Research introduced an exploration strategy that guides agents by maximizing the disagreement among an ensemble of deterministic forward dynamics models. This approach demonstrated improved robustness to stochasticity and achieved orders of magnitude higher sample efficiency in robotic manipulation tasks through differentiable policy optimization.
Key insights
- Ensemble disagreement robustly handles stochastic environments; models converge to the mean behavior, reducing intrinsic motivation once the average dynamics are understood, thus avoiding the 'noisy TV problem'.
- Formulating intrinsic reward as a differentiable objective enables direct policy optimization, significantly improving sample efficiency compared to traditional reinforcement learning methods.
- Training diverse models within the ensemble ensures that high disagreement reliably indicates genuinely novel or poorly understood parts of the state space.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.