Reproduce “Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying” on OpenResearch

About this paper

Jonathan Hecht, Lukas Arzoumanidis, Ziyue Li, Youness Dehbi

Researchers at HafenCity University Hamburg and TU Munich introduce and evaluate two novel multimodal contrastive learning architectures, MELT and SALT, for training implicit Earth embedding models using geographic coordinates to tie together unpaired satellite imagery, natural images, and text. The study demonstrates that while these architectures achieve competitive performance against strong baselines, the representational capacity of the location encoder itself, rather than modality diversity or increased pre-training data volume, primarily limits improvements in downstream spatial prediction tasks.

Key insights

• The representational capacity of the location encoder itself is the primary performance bottleneck; increasing the number of modalities beyond two or scaling pre-training data volume did not consistently improve downstream task performance.
• The contrastive objective, in its current NT-Xent formulation, might converge to a relatively low-complexity alignment, capturing commonalities but potentially discarding finer, modality-specific distinctions.
• The MELT architecture provides more stable training dynamics and smoother loss curves due to its simultaneous batch construction, making it a more robust foundation for future scaling compared to SALT's alternating approach.

Summary by alphaXiv.