Reproduce “Merging Models with Fisher-Weighted Averaging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michael Matena, Colin Raffel
Researchers from the University of North Carolina at Chapel Hill developed Fisher-weighted averaging, a principled method for merging neural network models based on a Bayesian perspective that leverages the diagonal Fisher information matrix. This approach consistently outperforms isotropic (unweighted) parameter averaging, improving trade-offs between in-distribution and out-of-distribution accuracy by approximately 1% in robust fine-tuning, and offers up to 6 times greater computational efficiency compared to gradient-based fine-tuning.
Key insights
- Simple parameter averaging can be understood as maximizing the joint likelihood of model posteriors approximated as isotropic Gaussians.
- Incorporating the diagonal Fisher information matrix into the Bayesian approximation of model posteriors provides a principled way to weigh the importance of individual parameters during merging.
- Model merging offers a flexible, non-iterative alternative to gradient-based fine-tuning for knowledge transfer, capable of combining knowledge from multiple sources and mitigating catastrophic forgetting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.