Reproduce “Detecting, Explaining, and Mitigating Memorization in Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxin Wen, Yuchen Liu, Chen Chen, Lingjuan Lyu
A unified framework is introduced for detecting, explaining, and mitigating memorization in text-to-image diffusion models. The approach leverages a novel detection metric derived from the magnitude of text-conditional noise prediction, demonstrating high efficiency and enabling both real-time detection and targeted mitigation strategies during inference and training.
Key insights
- Memorized prompts in diffusion models exhibit stronger text-conditional guidance, steering generation towards specific training images due to overfitting.
- The magnitude of text-conditional noise prediction can serve as an efficient and early signal for detecting memorization, usable from the first denoising step.
- Specific words or tokens within prompts can be identified as 'trigger tokens' responsible for initiating memorization, offering interpretable insights for mitigation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.