Reproduce “Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Odysseas Boufalis, Jorge Carrasco-Pollo, Joshua Rosenthal, Eduardo Terres-Caballero, Alejandro García-Castellanos
University of Amsterdam researchers introduced a symmetry-aware autoencoder, employing Scale Graph Metanetworks, to implicitly canonicalize neural networks by jointly addressing permutation and scaling symmetries. This framework established robust linear mode connectivity for merging Implicit Neural Representations and Convolutional Neural Networks, outperforming naive interpolation and explicit alignment methods, while offering efficient O(P) inference.
Key insights
- Leveraging the inherent equivariance of Scale Graph Metanetworks provides a principled method to jointly and systematically handle both permutation and scaling symmetries in neural networks.
- Implicitly canonicalizing networks into a symmetry-invariant latent space eliminates the need for computationally expensive, explicit combinatorial alignment procedures, improving efficiency.
- Training the autoencoder with a functional loss ensures that the learned canonical representation preserves the original network's function, enabling smooth linear interpolation paths for model merging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.