Reproduce “Diffusion-Based Neural Network Weights Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, Sung Ju Hwang
DIFFUSION-BASED NEURAL NETWORK WEIGHTS GENERATION (D2NWG) introduces a method for generating task-conditioned neural network weights using latent diffusion models. The approach demonstrates the ability to generate high-performing weights for various architectures, including large language models (LLMs), often outperforming meta-learning baselines without fine-tuning and significantly accelerating convergence when fine-tuned.
Key insights
- Diffusion models can effectively learn and generate complex distributions of neural network weights, conditioned on specific task data, moving beyond traditional transfer learning's reliance on pre-trained model selection.
- Optimal task-specific weights, including those for large language models, can be directly sampled, frequently outperforming fine-tuned meta-learning baselines without additional optimization or accelerating convergence significantly when fine-tuned.
- Techniques like Marchenko-Pastur distribution for layer selection and layer-wise chunking enable scalable weight generation for large models, addressing a critical limitation of previous hypernetwork approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.