Reproduce “e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings” on OpenResearch

About this paper

Haonan Chen, Sicheng Gao, Radu Timofte, Tetsuya Sakai, Zhicheng Dou

e5-omni introduces a lightweight, plug-and-play recipe for explicitly aligning embeddings from diverse modalities (text, image, audio, video, visual documents) using off-the-shelf VLM backbones. This approach establishes a more robust omni-modal embedding model, achieving an overall score of 66.4 on the MMEB-V2 benchmark and 37.7 Recall@1 on AudioCaps for text-audio retrieval.

Key insights

• Explicit alignment mechanisms, such as modality-aware temperature calibration and covariance alignment, are crucial for establishing a robust and geometrically consistent shared embedding space for omni-modal retrieval.
• Dynamically managing negative hardness during contrastive learning, via a curriculum schedule and debiasing techniques, enhances fine-grained discrimination and stabilizes optimization in multimodal settings.
• A lightweight, plug-and-play recipe can effectively adapt powerful pre-trained VLM backbones into strong omni-modal retrieval models without requiring complex architectural modifications or extensive retraining.
• Performance improvements from explicit alignment are complementary to scaling up VLM backbone sizes, indicating general applicability across different model capacities.

Summary by alphaXiv.