Reproduce “CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ankan Deria, Komal Kumar, Xilin He, Imran Razzak, Hisham Cholakkal, Fahad Shahbaz Khan, Salman Khan
CoME-VL combines language-aligned SigLIP2 features with spatially informed DINOv3 features. It selects and mixes features across layers, then uses position-aware cross-attention to align the streams before sending a visual representation to the language model. On PixMo, its average accuracy exceeds Molmo’s by 4.9 percentage points for understanding and 5.4 for grounding; inference averages 1.52 seconds per sample, versus 1.26 for Molmo.
Key insights
- Distinct visual encoders, like semantic-focused SigLIP and spatially-focused DINO, offer complementary information essential for comprehensive visual understanding and precise grounding.
- Carefully designed fusion, incorporating orthogonality constraints and entropy-guided layer selection, can effectively decorrelate features and mitigate redundancy from multi-layer visual representations.
- Geometry-aware alignment via RoPE-enhanced cross-attention enables efficient integration of heterogeneous visual token grids, crucial for maintaining computational efficiency while enhancing spatial reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.