Reproduce “LM-CMA: an Alternative to L-BFGS for Large Scale Black-box Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ilya Loshchilov
The LM-CMA algorithm provides a derivative-free optimization method for large-scale black-box functions, achieving competitive performance with L-BFGS when exact gradients are unavailable. It offers O(mn) memory and practical O(n) time complexity, enabling scalability to problems with millions of variables while maintaining robustness.
Key insights
- A derivative-free algorithm can achieve competitive or superior performance compared to L-BFGS in large-scale black-box optimization scenarios where L-BFGS relies on estimated gradients.
- Implicit reconstruction of the Cholesky factor from a limited history of direction vectors, combined with adaptive subset sampling, enables CMA-ES-like adaptation at a practical O(n) computational cost.
- Rademacher sampling and the Population Success Rule contribute to computational efficiency and robustness in high dimensions without relying on Gaussian assumptions or explicit gradient information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.