Reproduce “Mechanisms of Projective Composition of Diffusion Models” on OpenResearch

About this paper

Arwen Bradley, Preetum Nakkiran, David Berthelot, James Thornton, Joshua M. Susskind

The paper defines projective composition as preserving selected feature marginals of source models. It analyzes score combination, adding each concept’s score difference from a background model, and proves this achieves the target under Factorized Conditionals, with reverse-diffusion sampling guaranteed in that case. In CLEVR, an empty background yielded all three locations in 99 of 100 images; an unconditional background yielded none.

Key insights

• "Factorized Conditionals" provide theoretical conditions where linear score combination achieves Projective Composition and allows correct sampling from composed distributions.
• The chosen background distribution critically influences compositional success, particularly for out-of-distribution generalization, with "empty" backgrounds often outperforming unconditional ones in simple settings.
• While compositions can be theoretically sound in abstract feature spaces via smooth transformations, standard diffusion sampling is only guaranteed to work for orthogonal transformations, otherwise leading to potential failures.

Summary by alphaXiv.