Reproduce “LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ryugo Morita, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Andreas Dengel

Morita et al. introduce a training-free method called LGTM for text-to-image diffusion models, enabling explicit light direction control by manipulating the initial latent noise. This approach achieves over 77% accuracy in shadow placement, significantly outperforming baseline models, while preserving image quality and extending compatibility to structural guidance frameworks like ControlNet.

Key insights

  • Channel 1 of the initial latent noise in the VAE latent space of diffusion models is strongly correlated with global brightness and illumination direction, enabling direct manipulation for lighting control.
  • Fine-grained control over specific image attributes, like lighting, can be achieved by selectively manipulating specific channels within the initial latent noise without requiring additional training or model fine-tuning.
  • A simple linear gradient light mask, applied directly to the identified latent channel, is sufficient to effectively guide light direction and shadow placement in the generated output.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.