Reproduce “Test-time Adaptation for Cross-modal Retrieval with Query Shift” on OpenResearch

About this paper

Haobin Li, Peng Hu, Qianjun Zhang, Xi Peng, Xiting Liu, Mouxing Yang

Researchers from Sichuan University and Georgia Institute of Technology introduce Test-time adaptation for Cross-modal Retrieval (TCR), an online method that enhances the robustness of pre-trained vision-language models against query shift. TCR refines query predictions and rectifies intra- and inter-modality alignment issues, consistently outperforming existing test-time adaptation baselines on COCO-C, Flickr-C, and query-gallery shift benchmarks.

Key insights

• Query shift in cross-modal retrieval primarily causes diminished intra-modality uniformity (queries become less discriminative) and an amplified inter-modality gap (disrupted cross-modal alignment).
• Simply eliminating the modality gap is suboptimal; instead, rectifying the target modality gap towards a plausible estimated source modality gap is crucial for preserving pre-trained alignment.
• Retrieval tasks' predictions are inherently noisier than classification due to a vast number of candidates, necessitating adaptive weighting of predictions to prevent overfitting during test-time adaptation.

Summary by alphaXiv.