Reproduce “Vocabulary-free Image Classification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, Elisa Ricci

Researchers from the University of Trento and Fondazione Bruno Kessler introduce Vocabulary-free Image Classification (VIC), a new paradigm that assigns image classes from an unconstrained semantic space, addressing the limitation of static vocabularies in existing Vision-Language Models. Their training-free method, CaSED, leverages pre-trained VLMs and external vision-language databases to achieve higher classification accuracy and efficiency than powerful models like BLIP-2 on diverse benchmarks.

Key insights

  • External Vision-Language Databases (VLDs) are more effective than direct VQA-enabled VLM querying for representing and searching vast semantic spaces in vocabulary-free classification.
  • A combination of image-to-text and text-to-text similarity scores (using a caption centroid) is crucial for robust candidate scoring and mitigating the modality gap.
  • For retrieval-augmented systems, the quality of captions in external databases is more critical than their sheer quantity for optimal performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.