Reproduce “Blog: Survey of Optimizers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoran Xu
Ruoran Xu's survey redefines neural network optimizer design as a compositional stack rather than a search for a single algorithm, categorizing methods by temporal estimation, update geometry, horizon management, and representation/systems. The work identifies matrix-aware methods as a leading development and proposes a rigorous multi-dimensional evaluation protocol, revealing that optimizer performance is highly context-dependent.
Key insights
- Optimizer design has transitioned from selecting a single algorithm to engineering a 'compositional stack' of techniques, where components from different axes are combined for specific training scenarios.
- Matrix-aware optimization, which exploits the inherent matrix (or tensor) structure of neural network parameters, represents the most notable scientific development in 2025–2026.
- Optimizer performance is highly contingent upon various factors (model scale, data-to-parameter ratio, batch size, schedule, tuning budget), meaning there is no single, context-free optimizer that universally replaces AdamW.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.