Reproduce “FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

François Gardères, Shizhe Chen, Camille-Sovanneary Gauthier, Jean Ponce

Researchers from Louis Vuitton and Inria developed FACap, a large-scale, high-quality fashion dataset for fine-grained composed image retrieval, using an automated pipeline for annotation. They also introduced FashionBLIP-2, a specialized model that achieved competitive retrieval performance, with FACap annotations showing superiority in detail and saliency over existing datasets.

Key insights

  • Automated data generation, by leveraging Vision-Language Models (VLMs) for detailed captioning and Large Language Models (LLMs) for comparative text generation, can produce large-scale, high-quality, fine-grained datasets, addressing annotation bottlenecks in specialized domains.
  • Adapting general-domain VLMs with lightweight components, such as adapter modules in the image encoder and multi-head matching mechanisms, enables efficient domain transfer and enhances a model's ability to capture and utilize fine-grained visual and linguistic details.
  • The quality and granularity of training data significantly impact model performance in complex retrieval tasks, with datasets featuring more detailed and salient annotations leading to superior learning of domain-specific nuances.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.