Reproduce “FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition” on OpenResearch

About this paper

Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, Bolei Zhou

FreeControl is a training-free pipeline that enables fine-grained spatial control over any pre-trained text-to-image diffusion model with diverse condition types by using a two-stage approach. The method demonstrates superior image-text alignment and structure preservation compared to training-free baselines, achieving competitive performance with training-based methods while eliminating model-specific training.

Key insights

• Leading principal components of self-attention block features within a pre-trained T2I model provide a robust and consistent representation of semantic structure across various image modalities and time steps.
• Disentangling structure guidance from appearance guidance, with appearance derived from a concurrently generated unguided image, effectively prevents appearance leakage and ensures strong image-text alignment while maintaining high visual quality.
• Keys from the first self-attention layer in the U-Net decoder are optimal for feature extraction, offering superior controllability and better disentanglement of semantic components compared to other feature types or deeper network layers.

Summary by alphaXiv.