Reproduce “Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding” on OpenResearch

About this paper

Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, Zuxuan Wu

This research introduces a novel data synthesis pipeline and the SPEC benchmark to accurately diagnose the struggle of current Vision-Language Models with fine-grained compositional reasoning. It also presents an optimization strategy using hard-negative aware contrastive loss that substantially improves models' fine-grained understanding while preserving broad generalization capabilities.

Key insights

• Leading VLMs (CLIP, BLIP, FLAVA, CoCa) perform at near-random levels across most subsets of the SPEC benchmark, exposing a profound deficiency in fine-grained compositional understanding beyond simple object recognition.
• The developed data synthesis pipeline successfully generates visually controlled hard negatives, confirming that the models' poor performance on SPEC stems from inadequate compositional reasoning rather than data quality issues.
• Incorporating carefully constructed hard negative examples during fine-tuning can significantly improve a VLM's ability to discern subtle visual-linguistic differences without degrading its performance on broader zero-shot tasks.

Summary by alphaXiv.