Reproduce “DiffSim: Taming Diffusion Models for Evaluating Visual Similarity” on OpenResearch

About this paper

Yiren Song, Xiaokang Liu, Mike Zheng Shou

DiffSim re-purposes pre-trained diffusion models to evaluate visual similarity between images, introducing the Aligned Attention Score (AAS) to capture a broad spectrum of visual consistencies from low-level style to high-level semantics. The method achieved strong alignment with human perception on benchmarks like NIGHTS and outperformed baselines on new style and instance consistency benchmarks without fine-tuning.

Key insights

• Diffusion model U-Net attention layers encode detailed visual appearance and concept information, making them suitable for robust similarity evaluation without requiring additional training.
• The Aligned Attention Score (AAS) effectively addresses feature misalignment in dense representations, enabling more accurate and fine-grained similarity comparisons than simple distance metrics.
• DiffSim's configurable parameters, such as denoising timestep and U-Net layer, allow the metric to prioritize different types of similarity, from low-level style to high-level semantic or instance consistency.

Summary by alphaXiv.