Reproduce “Image-Caption Encoding for Improving Zero-Shot Generalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Yang Yu, Christopher Liao, Sathvik Ravi, Theodoros Tsiligkaridis, Brian Kulis
Image-Caption Encoding (ICE) improves zero-shot generalization of vision-language models for out-of-distribution data by re-ranking top predictions using the model's inherent captioning capabilities at inference time. This training-free approach achieved an average of 0.5% improvement in Top-1 accuracy across 15 OOD datasets, with gains up to 3% on challenging ones.
Key insights
- Generated captions from a VLM's internal decoder contain fine-grained spatial and semantic information that can effectively complement the image encoder's primary classification signal.
- A 'local search' mechanism within the Top-K image-based predictions, guided by caption information, efficiently improves Top-1 accuracy by re-ranking without requiring full re-evaluation across all classes.
- Dynamically adjusting the caption's influence based on the confidence of both image and caption distributions, alongside caption ensembling, leads to more robust and superior performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.