Reproduce “WeightCLIP: Aligning Datasets and Models for Weight Space Learning” on OpenResearch

About this paper

Aron Asefaw, Konstantinos Tzevelekakis, Damian Falk, Léo Meynent, Damian Borth

WeightCLIP introduces a method for aligning neural network weight representations with dataset representations in a shared latent space, addressing the lack of an explicit semantic reference frame in weight space learning. This approach facilitates improved model retrieval, dataset-guided weight generation, and efficient latent space refinement, achieving a 21% increase in kNN classification accuracy for model representations to their training datasets.

Key insights

• Explicitly incorporating dataset information through contrastive alignment significantly reshapes the neural network's latent space, organizing model representations into distinct, dataset-specific clusters.
• The dataset-aligned latent space enables controlled navigation and the generation of models tailored to specific datasets using small data prompts, even for out-of-distribution scenarios.
• Refinement conducted directly within the learned, structured latent space, leveraging a hyper-spherical constraint, can match or surpass the performance of traditional weight-space fine-tuning with comparable computational resources.

Summary by alphaXiv.