Reproduce “COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fanding Huang, Jingyan Jiang, Qinting Jiang, Hebei Li, Faisal Nadeem Khan, Zhi Wang
COSMIC enhances CLIP's test-time adaptation robustness by integrating multi-granular and cross-modal semantic information into a Dual Semantics Graph and employing Clique Guided Hyper-classes for adaptive querying, yielding substantial performance gains on out-of-distribution and cross-domain benchmarks while maintaining efficiency.
Key insights
- Integrating multi-granular (coarse CLIP and fine-grained DINOv2) and cross-modal (text and vision) semantic features within a graph structure significantly enriches the cache's knowledge and reduces noise.
- Leveraging higher-order relationships between classes through second-order graphs and identifying maximal cliques (hyper-classes) enables more robust and context-aware querying, filtering out noisy information.
- A well-designed training-free cache-based adaptation can surpass prompt-learning methods in efficiency and generalize effectively to diverse domain shifts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.