Reproduce “Model Fusion via Optimal Transport” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sidak Pal Singh, Martin Jaggi

A method is introduced for merging pre-trained neural networks into a single model by using optimal transport to align neurons across layers. This approach consistently outperforms direct weight averaging and enables one-shot, data-free fusion of models with varying architectures or those trained on non-i.i.d. data.

Key insights

  • Optimal Transport provides a robust mathematical framework to overcome the permutation invariance challenge when combining neural network parameters.
  • A greedy, layer-wise alignment of neurons using OT effectively fuses models by finding meaningful correspondences between their internal representations.
  • The approach allows for data-free fusion by aligning based on neuron weights, or with minimal unlabeled data using activations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.