Reproduce “jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, Han Xiao
Researchers at Jina by Elastic developed GELATO, a method for constructing multimodal embedding models that project text, image, audio, and video into a unified semantic space by aligning frozen, pre-trained encoders with minimal trainable projectors. The resulting `jina-embeddings-v5-omni` models achieve competitive multimodal performance against larger systems while training only 0.35% of the total parameters, with `jina-embeddings-v5-omni-small` scoring 54.04 across four modalities.
Key insights
- Freezing the text backbone and non-text encoders, and training only compact projector layers, significantly reduces computational overhead and training time while preserving the strong performance of the base text model.
- The GELATO architecture successfully creates an "interleaved geometry" in the embedding space, allowing different modalities to co-mingle and enabling unified semantic understanding across text, image, audio, and video.
- Modular construction via compact projectors allows for efficient extension of existing, high-performance text embedding models to multimodal capabilities without destabilizing their core competencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.