Reproduce “Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence” on OpenResearch

About this paper

Wenzhe Yin, Zehao Xiao, Pan Zhou, Shujian Yu, Jiayi Shen, Jan-Jakob Sonke, Efstratios Gavves

CS-Aligner, a framework developed at the University of Amsterdam, enhances vision-language alignment by integrating Cauchy-Schwarz divergence with mutual information, mitigating the modality gap and InfoNCE conflict. The method achieves competitive text-to-image generation quality, outperforms InfoNCE-based methods in retrieval tasks, and effectively leverages unpaired data and token-level relationships.

Key insights

• The inherent conflict between the alignment and uniformity components of InfoNCE loss exacerbates the modality gap in multimodal learning.
• Cauchy-Schwarz divergence provides a symmetric and robust metric to explicitly minimize the distance between vision and language feature distributions, complementing mutual information by ensuring distributional similarity.
• Leveraging unpaired data and fine-grained token-level relationships through distributional alignment significantly enhances model robustness and semantic precision beyond traditional pairwise, global approaches.

Summary by alphaXiv.