Reproduce “Vision-Language Grounding as Bidirectional Concept Correspondence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna
Researchers from the University of Washington and their affiliations introduced ConCor-1, a model that redefines vision-language grounding as bidirectional concept correspondence to jointly identify grounded text spans, instance-level image segments, and their alignments. The model consistently achieved superior F1-based metrics, including a 48% increase in JointF1 on the COCONut-PanCap dataset, demonstrating more effective many-to-many text-image alignment than prior methods.
Key insights
- Explicitly binding text spans to image regions via bridge tokens for bidirectional correspondence prediction significantly outperforms autoregressive language generation baselines for joint grounding.
- Pretrained visual and language representations from VLMs are crucial for robust object recognition, referential understanding, and effective cross-modal alignment with complex inputs.
- The proposed objective induces more spatially aligned, concentrated, and semantically meaningful phrase-level attention within the VLM backbone, indicating improved internal cross-modal alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.