Reproduce “CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning” on OpenResearch

About this paper

SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang, Byoung-Tak Zhang

Researchers from Seoul National University developed CS-CLIP, a framework that enhances vision-language models' compositional reasoning by generating precisely targeted hard negative captions. It leverages scene graphs, masked language models, and natural language inference to create these negatives, leading to state-of-the-art average accuracy of 88.6% on SugarCrepe with uniform performance across different reasoning types, while preserving general vision-language capabilities.

Key insights

• Controlled analysis revealed that existing compositionality-aware VLMs suffer from significant element-specific biases, often underperforming vanilla CLIP or exhibiting low confidence for attribute and relation changes.
• Precise structural guidance from scene graphs, combined with semantic verification from NLI, is crucial for generating high-quality hard negatives that effectively compel robust compositional reasoning.
• The quality and targeted nature of hard negatives are more impactful than sheer data volume, as CS-CLIP achieves strong performance with significantly fewer training samples than prior methods.

Summary by alphaXiv.