Reproduce “jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andreas Koukounas, Georgios Mastrapas, Sedigheh Eslami, Bo Wang, Mohammad Kalim Akram, Michael Günther, Isabelle Mohr, Saba Sturua and 2 more
JINA-CLIP-V2, developed by Jina AI, provides a multilingual multimodal embedding model for text and images, employing a multi-stage contrastive learning paradigm and Matryoshka Representation Learning. The model achieves competitive crossmodal retrieval performance and improved visual document understanding across 29 languages, while enabling up to 75% embedding truncation with minimal performance loss.
Key insights
- Multi-task learning with separate temperature parameters for each modality is superior to a unified batch training approach due to information asymmetry between visual and textual data.
- Progressive increases in image resolution and inclusion of specialized visually rich document data significantly enhance the model's visual document understanding capabilities.
- Matryoshka Representation Learning effectively allows embedding truncation up to 75% (from 1024 to 256 dimensions) with less than 1% performance degradation across various tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.