Reproduce “Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights” on OpenResearch

About this paper

Konstantin Schürholt, Boris Knyazev, Xavier Giró-i-Nieto, Damian Borth

Researchers extended hyper-representation learning to enable generative sampling of functional neural network weights by introducing Layer-Wise Loss Normalization (LWLN). This approach yields models that offer better initialization, faster training, and improved performance in transfer learning and ensemble tasks, even for unseen architectures.

Key insights

• Layer-Wise Loss Normalization (LWLN) is critical for ensuring that decoded neural networks are functional and perform significantly above random chance.
• The learned hyper-representation latent space, particularly with LWLN, exhibits remarkable smoothness, enabling effective interpolation and sampling of functional models.
• Hyper-representations can generalize effectively, generating functional initializations for models from unseen tasks and even novel architectural variations not present in the original training data.

Summary by alphaXiv.