Reproduce “SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski
SOCO introduces a benchmark for testing whether vision foundation models can match semantically corresponding object parts across images, distinguishing basic concept recognition from repeated-part disambiguation and cross-category transfer. Across 100 categories and more than one million correspondence pairs, dense self-supervised features generally outperform language-aligned models, while SOCO scores correlate more strongly with dense vision tasks than ImageNet classification accuracy.
Key insights
- Models consistently perform better at recognizing a part concept than at selecting the correct repeated instance, with another drop when transferring the concept across categories; DINOv2, for example, scores 78.9% CC, 60.4% SOC, and 54.9% Cross-SOC.
- Dense self-supervised representations are generally better suited to visual correspondence than globally aligned image-text models, and model rankings differ between semantic matching and geometric disambiguation.
- LVLMs benefit substantially from textual keypoint descriptions but remain weaker than dense visual features for direct cross-image correspondence, indicating that language grounding does not guarantee fine-grained visual matching.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.