Reproduce “Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models” on OpenResearch

About this paper

Rowan Bradbury, Dazhi Zhong

Researchers at the Bradbury Group developed Pixel-Equivalent Latent Compositing (PELC) and DecFormer, a 7.7M-parameter transformer, to address artifacts from linear latent blending in diffusion models. This approach consistently reduces error metrics around mask edges by up to 53% and halves perceptual error (LPIPS), enabling more geometrically consistent and artifact-free image editing.

Key insights

• The widespread practice of linear latent mixing in LDM image editing is a foundational flaw, as VAEs' nonlinearity means latent-space linearity does not map to pixel-space linearity.
• Achieving pixel-equivalent latent operations requires both channel-adaptive blend weights and a nonlinear residual correction to account for complex VAE latent space curvature.
• Addressing latent blending before denoising provides a more geometrically consistent and artifact-free base for LDM operations, reducing the burden on the diffusion model to correct VAE-induced errors.

Summary by alphaXiv.