Reproduce “Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuwen Yu, Zhanxuan Hu, Yi Zhao, Yonghang Tai, Huafeng Li
The GPUA framework aligns Vision-Only Foundation Models with Vision-Language Foundation Models by learning an unsupervised, geometry-preserving orthogonal transformation. This approach effectively integrates fine-grained visual discrimination with language-grounded semantics, leading to performance improvements in zero-shot classification and open-vocabulary segmentation tasks.
Key insights
- Unsupervised visual representation alignment can be effectively achieved by treating it as a cross-modal translation problem, leveraging shared structural regularities between heterogeneous foundation models.
- Coupling geometric structural coherence from Vision-Only Foundation Models with semantic consistency from Vision-Language Foundation Models is crucial for robustly estimating correspondence without explicit labels.
- Preserving the intrinsic feature geometry through orthogonal transformations and mitigating hubness with a specialized loss function are key to learning a stable, discriminative, and generalizable aligned embedding space.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.