Reproduce “EOE: Evolutionary Optimization of Experts for Training Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yingshi Chen
The Evolutionary Optimization of Experts (EOE) framework facilitates training large language models (LLMs) on a single consumer-grade GPU by partitioning the model into distinct, sequentially trained 'experts' and incorporating evolutionary optimization principles. This approach allows for training of models like GPT-2 XL on 24GB GPUs, achieving up to 50 times faster throughput and producing a compact inference model that is roughly one-fifth the size of the full model.
Key insights
- Sequentially training distinct, complete "experts" of an LLM, rather than simultaneously, drastically reduces GPU memory consumption during training.
- Integrating evolutionary optimization principles can effectively guide the learning process by transferring knowledge from a "best" expert to a population of experts.
- A single compact expert, derived from an evolutionarily optimized population, can achieve competitive performance with a significantly reduced model size for deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.