Reproduce “Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonas Loos, Lorenz Linhardt
An analysis of U-Net-based Stable Diffusion model representations revealed three inherent properties: a linearly extractable positional embedding with over 90% prediction accuracy in lower blocks, content-agnostic corner token similarities, and localized high-norm anomalous activations present in approximately 25% of images. These findings across SD-1.5, SD-2.1, and SD-Turbo models suggest potential interferences for feature-based downstream applications.
Key insights
- Lower U-Net blocks contain a strong, linearly extractable positional encoding, which could confound semantic similarity in downstream tasks.
- Specific artifacts exist, including content-agnostic high cosine similarities among corner tokens and localized high-norm activations, which may not correspond to semantic content.
- These identified phenomena are consistent across various Stable Diffusion architectures, indicating they are general properties of this model family.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.