Reproduce “CLIP-GCD: Simple Language Guided Generalized Category Discovery” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rabah Ouldnoughi, Chia-Wen Kuo, Zsolt Kira

CLIP-GCD, a framework developed at Georgia Tech, improves Generalized Category Discovery (GCD) by combining CLIP's visual features with semantically rich textual context retrieved from large text corpora. The model consistently surpassed prior methods, achieving up to 6.7% higher accuracy on all classes and 11.6% higher accuracy on unknown classes on average across generic datasets, and showed marked improvements on fine-grained and domain adaptation benchmarks.

Key insights

  • Language-guided features (combining image and retrieved text) consistently and substantially outperform image-only features in GCD, demonstrating the critical role of semantic context.
  • Fine-tuning the CLIP model on the target dataset yields significant performance improvements for GCD, challenging the assumption that CLIP should always be used frozen.
  • The quality and diversity of the text description database, along with the optimal number of retrieved captions, influence performance, indicating the importance of rich contextual data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.