Reproduce “How Compositional Generalization and Creativity Improve as Diffusion Models are Trained” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alessandro Favero, Antonio Sclocchi, Francesco Cagnetta, Pascal Frossard, Matthieu Wyart
The authors study how diffusion models learn hierarchical structure by grouping patterns with similar statistical contexts and reusing them at longer ranges. In a random hierarchy grammar, rule learning proceeds from lower to deeper levels, and sample complexity follows cubic scaling in synonym count at depth two. Natural-data results are compatible, but establish neither grammar equivalence nor semantic coherence.
Key insights
- Diffusion models learn compositional rules in a hierarchical, staged manner, acquiring lower-level (local) rules before higher-level (global) rules.
- The sample complexity for learning hierarchical rules scales polynomially with data dimension (specifically, as m^(ℓ+1) for level ℓ), overcoming exponential scaling when data has latent hierarchical structures.
- The core learning mechanism involves clustering features with statistically similar contexts, enabling the model to infer abstract, hierarchical representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.