Reproduce “VN-MTEB: Vietnamese Massive Text Embedding Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Loc Pham, Tung Luu, Thu Vo, Minh Nguyen, Viet Hoang
VN-MTEB extends the Massive Text Embedding Benchmark to Vietnamese with 41 translated datasets covering six embedding tasks, using language detection, multilingual similarity checks, and LLM-based quality filtering. Among 18 evaluated models, multilingual-e5-large-instruct achieved the highest overall average score at 67.99%, while Vietnamese-specific models generally trailed multilingual systems.
Key insights
- Automated filtering retains roughly 53.4%–72.0% of samples depending on task, with lower retention on technically demanding datasets such as SciDocs and SICK-R.
- Multilingual models outperform the tested Vietnamese-specific models overall; multilingual-e5-large-instruct leads at 67.99%, followed by e5-Mistral-7B-instruct at 67.67%.
- Instruction tuning and larger parameter counts are generally associated with stronger scores, whereas the apparent advantage of RoPE over absolute positional encoding is confounded by other architectural and training differences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.