Reproduce “Graph Neural Networks for Learning Equivariant Representations of Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, David W. Zhang
Researchers introduce a unified framework for learning permutation-equivariant representations of neural networks by modeling them as "neural graphs." This approach, leveraging adapted Graph Neural Networks and Transformers, enables processing of heterogeneous neural network architectures and consistently outperforms prior state-of-the-art methods across various meta-learning tasks.
Key insights
- Representing neural networks as computational graphs provides a principled and elegant solution to the permutation symmetry problem by intrinsically aligning with GNN and Transformer capabilities.
- The neural graph approach enables a single meta-learning model to effectively process and generalize across highly heterogeneous neural network architectures, a significant advancement over fixed-architecture methods.
- Enriching node and edge features with 'probe features' (intermediate activations), positional embeddings, and non-linearity embeddings substantially improves the model's ability to capture the target network's behavior and structure.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.