Reproduce “DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sung-Hoon Yoon, Hoyong Kwon, Changgyoon Oh, Kuk-Jin Yoon
The DINOde framework, developed by researchers from DGIST and KAIST, introduces a continuous, Ordinary Differential Equation (ODE)-based approach for aligning self-supervised visual features with CLIP text embeddings for open-vocabulary semantic segmentation. This method preserves feature space geometry, achieving state-of-the-art performance on 6 out of 8 benchmarks and an average 1.9%p mIoU improvement over the Talk2DINO* baseline using the same DINOv3 backbone.
Key insights
- Continuous, geometry-preserving transformations via ODEs are more effective for cross-modal alignment than discrete projections, mitigating semantic entanglement and maintaining topological relationships.
- Integrating both local (DINO patch tokens via STF) and global (DINO [CLS] token via GCF) visual context with text embeddings leads to more robust and accurate open-vocabulary semantic segmentation.
- Velocity Tangent Projection (VTP) is crucial for maintaining the hyperspherical geometry of feature spaces during ODE integration, contributing to stability and accuracy in manifold transitions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.