Reproduce “Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aditya Kumar Akash, Sixu Li, Nicolás García Trillos
This research introduces a unified mathematical framework for neural network model fusion based on Wasserstein and Gromov-Wasserstein barycenters, applicable to diverse architectures including recurrent networks. The approach provides empirical evidence for the linear mode connectivity of SGD solutions after permutation, demonstrating low-loss linear paths across various network types.
Key insights
- Optimal transport-based sparse couplings effectively align neurons, enabling robust model fusion across diverse architectures and heterogeneous data distributions.
- Explicitly modeling hidden-to-hidden recurrent connections through the GWB framework is crucial for accurate and effective fusion of RNNs and LSTMs.
- The learned sparse couplings effectively permute local minima, revealing low-barrier linear paths between different SGD solutions and providing strong empirical support for permutation-invariant linear mode connectivity across various network types, including recurrent networks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.