Reproduce “Re-basin via implicit Sinkhorn differentiation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, Marco Pedersoli
Researchers developed a differentiable re-basin method leveraging implicit Sinkhorn differentiation to find optimal permutation alignments between neural networks. This approach significantly improved linear mode connectivity between model solutions and, when applied to continual learning, achieved the highest average accuracy (78.14%) and lowest forgetting (0.12%) on Rotated Mnist, and competitive performance on Split Cifar100 (51.34% accuracy, 0.07% forgetting).
Key insights
- Re-basin can be effectively made differentiable by leveraging optimal transport with implicit Sinkhorn differentiation, allowing for simultaneous optimization of all permutation matrices.
- Differentiable re-basin leads to more optimal permutations and significantly lower loss barriers between independent model solutions, confirming stronger linear mode connectivity.
- A differentiable re-basin mechanism can be successfully applied to continual learning, providing a flexible way to merge models and mitigate catastrophic forgetting by balancing plasticity and stability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.