Reproduce “Modular Energy Steering for Safe Text-to-Image Generation with Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaoteng Tan, Zikui Cai, M. Salman Asif
This research introduces a training-free and dataset-free inference-time steering framework for text-to-image models, utilizing frozen vision-language foundation models as semantic energy estimators to guide generations away from unsafe content. The method effectively suppresses NSFW and specific identities across various diffusion models while largely preserving benign image quality.
Key insights
- Pretrained vision-language foundation models inherently possess rich semantic knowledge about harmful concepts, allowing them to serve as effective, 'training-free' safety energy estimators.
- Framing safety as an energy-based steering problem enables a principled and differentiable mechanism for control that is compatible with continuous-time generative models like diffusion and flow-matching models.
- By estimating clean latents at intermediate generation steps, the framework provides visually coherent signals to foundation models, allowing for precise and early intervention in the generative process.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.