Reproduce “ESSA: Evolutionary Strategies for Scalable Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daria Korotyshova, Boris Shaposhnikov, Alexey Malakhov, Alexey Khokhulin, Nikita Surnachev, Kirill Ovcharenko, George Bredis, Alexey Gorbatovski and 2 more
A gradient-free method, ESSA, for large language model alignment, utilizes low-rank adaptation and singular value decomposition with evolutionary strategies. This approach achieves competitive performance against gradient-based methods while significantly improving scalability and resource efficiency.
Key insights
- A compact, SFT-seeded singular-value parameterization effectively supports population-based gradient-free search for online policy optimization in large language models.
- Decoupling the policy optimization from gradient computation leads to substantial improvements in parallel scalability and resource efficiency, enabling native compatibility with aggressive weight quantization (e.g., INT4).
- The SFT-derived singular-value basis contributes to faster convergence and higher accuracy compared to a random basis, though the population-based search itself is a primary driver of performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.