Reproduce “CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space” on OpenResearch

About this paper

Sohwi Lim, Lee Hyoseok, Jungjoon Park, Tae-Hyun Oh

KAIST researchers introduced CLAY, a training-free method that dynamically modulates visual similarity in pretrained Vision-Language Model embedding spaces based on textual conditions. This approach allows for efficient, adaptive image retrieval, achieving higher accuracy than existing methods in both single and multi-conditional settings on various datasets while significantly reducing computational overhead.

Key insights

• The intrinsic structure of VLM embedding spaces can be effectively repurposed to define adaptive, text-conditional similarity metrics.
• Manifold-aware projection combined with orthonormal rotation is crucial for accurately preserving relationships and effectively modulating visual features within the hyperspherical VLM embedding space.
• Decoupling textual conditioning from visual feature extraction enables a highly efficient symmetric conditional similarity formulation, addressing the computational overhead of previous training-based methods.

Summary by alphaXiv.