Reproduce “A Night at the LLM Zoo: Data Conditioned Flow Matching for Generating Pretrained LLM Weights” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Scott Biggs, Bedionita Soro, Saumya Gupta, Bruno Andreis, Sung Ju Hwang

The authors train GPT-2 populations across mixtures of five text domains, then learn a conditional flow that generates model weights for a requested mixture without optimizer steps. On held-out mixtures, the 51-million-parameter Mini model has 4.05% five-domain perplexity deviation, versus 5.75% for matched-blinding nearest-neighbor retrieval. The advantage holds on interior mixtures, not all vertices; the PCA basis includes held-out weights.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.