Reproduce “Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianyi Li, Zhiqiang Shen

The Dual Learned Matching (LMC-DM) framework scales linear mode connectivity to billion-parameter Transformer models by formulating and exploiting comprehensive architectural symmetries. This method significantly reduced loss barriers between independently trained models, enabling effective merging via linear interpolation and achieving near-zero loss paths for medium-sized language models, alongside low barriers for billion-parameter models.

Key insights

  • Linear mode connectivity is achievable in billion-parameter Transformers when comprehensive architectural symmetries are explicitly identified and leveraged.
  • Dual optimization, where learnable symmetry transformations are applied to both models being merged, provides a more favorable optimization landscape and consistently yields significantly lower loss barriers compared to single-sided approaches.
  • Explicitly modeling and resolving a wide range of Transformer-specific symmetries, especially those within attention QK/OV circuits, is crucial for reducing interpolation loss barriers between large, independently trained models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.