Reproduce “Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Denis Coquenet, Clément Rambour, Emanuele Dalsasso, Nicolas Thome
A multitask fine-tuning strategy for Vision-Language Foundation Models (VLFMs) enhances their capability for fine-grained classification, attribute detection, and localization, achieving substantial performance gains on the CUB200-2011 dataset, with up to 91.20% top-1 accuracy and 74.85% localization mAP. The method also mitigates annotation burden by leveraging other multimodal foundation models for pseudo-label generation.
Key insights
- Multitask fine-tuning significantly improves the fine-grained understanding of VLFMs, enabling them to move beyond global understanding to granular visual comprehension for both attribute detection and localization.
- A dedicated projection layer for image patch embeddings is crucial for aligning local visual features with attribute embeddings, unlocking effective fine-grained localization capabilities.
- Utilizing other MMFMs for pseudo-label generation offers a viable and cost-effective method to relax the annotation burden for localization tasks, achieving competitive results compared to expert annotations.
- The positive/negative prompt formulation for attribute detection enhances model generalization, allowing for improved detection of previously unseen attributes without additional training data for those specific attributes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.