Reproduce “GigaEmbeddings: Efficient Russian Language Embedding Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Egor Kolodin, Daria Khomich, Nikita Savushkin, Anastasia Ianina, Fyodor Minkin
GigaEmbeddings introduces an efficient Russian language embedding model built upon the GigaChat-3B decoder-only Large Language Model. The model achieved an average score of 69.1 on the ruMTEB benchmark, establishing new state-of-the-art performance for Russian text embeddings and demonstrating a scalable paradigm for lower-resource languages.
Key insights
- Decoder-only LLMs can be effectively adapted as universal text encoders, demonstrating competitive performance with traditional encoder-only architectures for embedding tasks.
- LLM-generated synthetic data provides a scalable and robust solution for addressing data scarcity in lower-resource languages, reducing the dependency on extensive manual annotation.
- Architectural optimizations, such as pruning 25% of the LLM's deeper layers, can significantly enhance computational efficiency and reduce inference latency with negligible impact on embedding quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.