Reproduce “VladVA: Discriminative Fine-tuning of LVLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Brais Martinez, Georgios Tzimiropoulos
VladVA, developed by Samsung AI Cambridge, is a discriminative fine-tuning framework that transforms generative Large Vision-Language Models into highly capable discriminative models. This method enables LVLMs to achieve state-of-the-art performance on image-text retrieval benchmarks such as Flickr30k and MS-COCO, and demonstrate advanced compositional understanding on SugarCrepe and Winoground, often outperforming larger, contrastively-trained models.
Key insights
- Generative Large Vision-Language Models can be effectively transformed into highly capable discriminative models through specific fine-tuning.
- A hybrid training strategy combining image-text contrastive loss (for short captions) and autoregressive loss (for long captions) is crucial for acquiring both strong discrimination and deep language understanding.
- Parameter-efficient adaptation techniques, such as soft prompting and LoRA, enable effective fine-tuning of large models while maintaining computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.