Reproduce “Celo: Training Versatile Learned Optimizers on a Compute Diet” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhinav Moudgil, Boris Knyazev, Guillaume Lajoie, Eugene Belilovsky
Celo introduces a learned optimizer demonstrating strong meta-generalization across diverse, unseen tasks with a meta-training budget of only 24 GPU hours. This system surpasses 15 hand-crafted and 4 learned optimizers, achieving a Final Loss IQM of 1.20 and a Speedup IQM of 1.86, making advanced learned optimization more accessible.
Key insights
- Task augmentation significantly enhances meta-generalization, particularly when applied during the training of the parameter update rule.
- Decoupled, two-stage training of the learned update rule and scheduler is critical for preventing meta-overfitting and improving generalization under compute constraints.
- A learned scheduler with an exponential output functional form and reliance on task-agnostic global features is essential for adaptive step-size control and robust generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.