Reproduce “Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, Noel E. O'Connor
Researchers at ML Labs, Dublin City University developed a method to automatically generate visually descriptive text (VDT) using GPT-4, which, when incorporated into CLIP, improved zero-shot classification accuracy across 12 datasets by an average of 2%. Additionally, a novel self-attention based adapter, CLIP-A-self, leveraged this information for superior and more efficient few-shot transfer in challenging base-to-new settings.
Key insights
- Visually descriptive text, even when generated by a large language model like GPT-4, significantly enhances CLIP's zero-shot transfer capabilities, particularly for fine-grained classification tasks.
- Large Language Models can function as automated 'domain experts,' bridging semantic knowledge with visual understanding to generate structured, class-conditional visual descriptions.
- A simple, learnable self-attention mechanism can effectively leverage an ensemble of VDT sentences, dynamically prioritizing discriminative visual cues for robust and computationally efficient few-shot learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.