Reproduce “How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Max Weltevrede, Moritz A. Zanger, Matthijs T.J. Spaan, Wendelin Böhmer
This research theoretically analyzes how policy distillation enhances reinforcement learning agent generalization, identifying that increasing the ensemble size and the diversity of distillation data improves performance. Ensembles of distilled policies, particularly when trained on diverse datasets, consistently generalized better than single policies and even surpassed the original teacher agent in complex environments like Minigrid Four Rooms.
Key insights
- Generalization performance of distilled policies is theoretically bounded, showing direct improvement with larger ensemble sizes (following a 1/√N relationship) and increased diversity of distillation data.
- Ensembles of distilled policies implicitly leverage underlying data symmetries, becoming more invariant to these symmetries, which is a key mechanism for generalizing to unseen contexts.
- Distilling on more diverse datasets, even if collected from existing training contexts, is crucial for fostering better generalization capabilities in student policies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.