Reproduce “Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction” on OpenResearch

About this paper

Konstantin Schürholt, Dimche Kostadinov, Damian Borth

A method learns task-agnostic representations directly from neural network weights using self-supervised learning, enabling accurate prediction of diverse model characteristics like test accuracy, generalization gap, and hyperparameters. These "hyper-representations" generalize to out-of-distribution model populations and outperform existing baselines.

Key insights

• Permutation augmentation, which exploits the symmetries of neural network weights, is crucial for achieving generalization and robust representation learning, especially under high compression.
• Attention-based architectures, particularly when combined with a dedicated compression token, effectively process and aggregate information from the complex, high-dimensional weight space into concise representations.
• A self-supervised loss combining both reconstruction and contrastive objectives provides the best balance, fostering good compression while producing well-structured and distinctive hyper-representations.

Summary by alphaXiv.