Reproduce “An Optimisation Framework for Unsupervised Environment Design” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nathan Monette, Alistair Letcher, Michael Beukman, Matthew T. Jackson, Alexander Rutherford, Alexander D. Goldie, Jakob N. Foerster
Researchers at FLAIR, University of Oxford, alongside collaborators from UC Irvine, introduce an optimization framework for Unsupervised Environment Design (UED) that reformulates the problem as nonconvex-strongly-concave, enabling provable convergence guarantees. The framework, called NonConvex-Concave (NCC) Optimization, enhances agent robustness and generalizability, demonstrating improved performance on Minigrid, XLand-Minigrid, and Craftax benchmarks.
Key insights
- Reformulating the Unsupervised Environment Design (UED) problem with an entropy regularization term for the adversary's distribution transforms the objective into a nonconvex-strongly-concave structure, which is more amenable to stable optimization.
- Two-timescale stochastic gradient descent-ascent algorithms, like the proposed NonConvex-Concave (NCC) Optimization, can provide provable convergence guarantees for UED in zero-sum settings.
- Generalizing the 'learnability' score function allows UED to effectively identify and prioritize levels of intermediate difficulty, fostering more efficient agent training across arbitrary deterministic environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.