Reproduce “Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, Luke Zettlemoyer

The Branch-Train-Merge (BTM) paradigm enables embarrassingly parallel training of Expert Language Models (ELMs) by specializing them to different domains and combining them into an ELMFOREST. This approach consistently outperforms monolithic language models and achieves substantial training speedups, reducing computational costs for large-scale language model development.

Key insights

  • The 'seed' training phase is essential for BTM's success, establishing shared representations crucial for effective parameter averaging and ensemble performance.
  • Embarrassingly parallel training of domain-specific expert models, without cross-node synchronization, yields substantial efficiency gains and allows for modularity.
  • Performance improvements stem from genuine domain specialization, as experiments with random data splits resulted in significantly worse performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.