Reproduce “Bayesian Nonparametrics for Offline Skill Discovery” on OpenResearch

About this paper

Valentin Villecroze, Harry J. Braviner, Panteha Naderian, Chris J. Maddison, Gabriel Loaiza-Ganem

Researchers at Layer 6 AI and the University of Toronto introduced a nonparametric framework for offline skill discovery, leveraging modern variational inference and a GEM prior to automatically determine the number of skills from expert trajectories. This approach achieved competitive or superior agent performance in Atari environments while removing the need for manual hyperparameter tuning of skill count.

Key insights

• Bayesian nonparametric methods, particularly those based on the Dirichlet process and stick-breaking processes, can effectively infer the intrinsic number of skills from expert trajectories without prior specification.
• Integrating modern variational inference techniques and continuous relaxations (Gumbel-Softmax) into offline option discovery leads to more effective and robust skill learning compared to older EM-based or variational approaches.
• The concept of dynamically incrementing the number of active options based on their utilization provides a practical way to manage the infinite options implied by nonparametric priors, enabling tractable variational inference.

Summary by alphaXiv.