Reproduce “Celo2: Towards Learned Optimization Free Lunch” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhinav Moudgil, Boris Knyazev, Eugene Belilovsky
CELO2 introduces a learned optimization algorithm meta-trained in just 4.5 GPU hours, demonstrating stable and performant generalization to billion-parameter models and diverse tasks, including language modeling and vision, often surpassing VeLO and matching well-tuned AdamW baselines.
Key insights
- Full RMS normalization of the learned MLP's output is critical for forcing a task-invariant update rule and achieving robust generalization to large-scale, out-of-distribution tasks.
- Decoupling the step-size from the learned update rule, making it a tunable hyperparameter, simplifies the learned function and significantly improves its generalization to large-scale problems.
- Task augmentation during meta-training and the strategic integration of techniques like orthogonalization or specific update rules for 1D parameters are crucial for enhancing performance and stability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.