Reproduce “Hyper-Representations: Learning from Populations of Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Konstantin Schürholt

Konstantin Schürholt's dissertation introduces "hyper-representations," a self-supervised framework from the University of St.Gallen for learning compact embeddings directly from neural network weights. This method facilitates analysis of trained models and the generation of new functional models, achieving competitive performance for discriminative tasks and outperforming baselines in generative tasks across various architectures.

Key insights

  • Neural network weights are structured and contain discernible information about model properties, making them suitable for representation learning.
  • Explicitly addressing weight space symmetries, particularly through permutation augmentation, is essential for stable and generalizable representation learning.
  • Generating functional models from learned weight representations requires careful reconstruction, with Layer-Wise Loss Normalization (LWLN) being critical for balancing layer-wise quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.