Reproduce “Personalizing Retrieval using Joint Embeddings or "the Return of Fluffy"” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bruno Korbar, Andrew Zisserman

A trainable mapping network, π-map, enables personalized image and video retrieval by translating localized object instance embeddings into unique text tokens for use with frozen Vision-Language Models. This approach significantly improves personalized search performance, requiring minimal data and computational resources, and allows for robust compound queries combining specific instances with natural language descriptions.

Key insights

  • Effective personalization for Vision-Language Models can be achieved efficiently by using a lightweight mapping network (π-map) on frozen VLM encoders, preserving general knowledge and reducing computational overhead.
  • Combining localized visual features with rich, automatically generated textual descriptions from Large Language Models significantly enhances the robustness and semantic richness for instance-specific learning.
  • A specialized training objective involving contrastive losses in both text and image embedding spaces, along with dedicated conditioning vectors, effectively bridges the modality gap for unique instance identification.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.