Reproduce “Learning Model Representations Using Publicly Available Model Hubs” on OpenResearch

About this paper

Damian Falk, Konstantin Schürholt, Konstantinos Tzevelekakis, Léo Meynent, Damian Borth

Researchers at the AIML Lab, University of St.Gallen, developed a method for learning unified representations directly from highly heterogeneous neural network models available on public hubs like Hugging Face. This approach enabled the generation of functional models across a wide range of architectures, datasets, and even different modalities, frequently surpassing the performance of models initialized from traditional, curated model zoos and improving training efficiency.

Key insights

• Masked Loss Normalization (MLN) effectively enables stable training and accurate reconstruction of heterogeneous neural network weights, focusing on signal values and avoiding biases from padding in diverse architectures.
• Efficient model weight processing via dense tokenization and sinusoidal positional encodings is crucial for scaling WSL to vast, diverse public model collections without incurring prohibitive memory or computational overhead.
• The diversity and quantity of models sourced from public hubs like Hugging Face are fundamental for learning generalizable and robust weight space representations, outperforming representations derived from homogeneous model zoos.

Summary by alphaXiv.