Reproduce “Foundation Models Boost Low-Level Perceptual Similarity Metrics” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Abhijay Ghildyal, Nabajeet Barman, Saman Zadtootaghaj

Utilizing intermediate features from pre-trained foundation models, particularly DINOv1, leads to highly accurate and robust training-free metrics for low-level perceptual similarity, surpassing both traditional and learned state-of-the-art methods in evaluating image quality against human perception.

Key insights

  • Intermediate features from foundation models capture more relevant, localized visual information crucial for low-level perceptual similarity tasks compared to their final, high-level semantic embeddings.
  • Foundation models, especially DINOv1, inherently provide features that yield superior robustness to common geometric misalignments (translation, dilation, rotation) than older CNN-based backbones and other learned metrics.
  • Effective perceptual similarity metrics can be derived from powerful pre-trained foundation models without requiring any additional training or fine-tuning on human perceptual judgment datasets.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.