Reproduce “Unimodal Augmentation-based SSL Objectives Induce Cross-Modal Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zohar Mashiah, Nir Ben-Ari, Ronen Talmon, Uri Shaham
The authors show that independently trained augmentation-based self-supervised models can develop aligned semantic neighborhoods across modalities. InfoNCE links this effect to augmentations that alter view-specific details while preserving semantic signal, and theory connects unimodal loss to CKA under assumptions. In pretrained models, lower loss associates with alignment, including Spearman ρs=0.88 and Pearson r=0.92 on COCO; this measures alignment, not task performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.