Reproduce “microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sathira Silva, Eman Ali, Chetan Arora, Muhammad Haris Khan

microCLIP is an unsupervised framework that enhances CLIP's capabilities for fine-grained image classification by fusing local visual cues from patch tokens with global context. The method achieved an average accuracy of 68.68% across 13 fine-grained datasets, outperforming prior unsupervised adaptation techniques by 2.90%.

Key insights

  • CLIP's pre-trained patch tokens contain valuable latent fine-grained visual signals that can be effectively extracted and utilized for detailed classification through saliency-guided attention pooling.
  • A symmetric fusion of both global and fine-grained visual tokens is crucial for robust pseudo-labeling, significantly improving performance in fine-grained classification by leveraging complementary information.
  • Progressive pseudo-label refinement benefits significantly from dynamically aggregating both stable pre-trained knowledge and evolving learned features, demonstrating that neither alone is sufficient for optimal unsupervised adaptation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.