Reproduce “$β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fatimah Zohra, Chen Zhao, Hani Itani, Bernard Ghanem
β-CLIP addresses limitations in fine-grained and long-text understanding for vision-language models by employing hierarchical text decomposition, text-conditioned visual feature selection, and a β-contextualized contrastive alignment loss. This framework improves fine-grained retrieval performance by recovering 55% of the gap to region-supervised methods and establishes a new state-of-the-art on the Urban1K long-text benchmark.
Key insights
- The β-CAL objective, especially with Cross-Entropy, significantly improves fine-grained discrimination by enabling tunable contextualization among hierarchical textual and visual representations.
- A tunable parameter β allows explicit control over the trade-off between strict query-specific matching and broader intra-image contextualization, critical for learning from semantically overlapping hierarchical data.
- Cross-Entropy (CE) loss is more effective for sharpening fine-grained alignments, while Binary Cross-Entropy (BCE) loss consistently yields better performance for long-text and coarse-grained retrieval tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.