Reproduce “Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qingcheng Zeng, Yuheng Lu, Zeqi Zhou, Heli Qi, Puxuan Yu, Fuheng Zhao, Hitomi Yanaka, Weihao Xuan and 1 more
A multi-institutional team systematically evaluated the impact of code-switching on information retrieval systems, demonstrating consistent performance degradation across diverse IR architectures, models, and language pairs. The research, which developed new human-annotated (CSR-L) and LLM-generated (CS-MTEB) benchmarks, found that existing retrievers, including strong multilingual models, exhibit significant challenges in maintaining retrieval quality with mixed-language queries, with performance drops up to 15 points for English-centric models.
Key insights
- Code-switching consistently degrades retrieval quality across all evaluated IR methods and language pairs, impacting even robust multilingual models.
- English-centric models show significant embedding space divergence between monolingual and code-switched queries, while multilingual models, though more resilient, still exhibit a performance gap.
- Lexicon-based vocabulary expansion offers only partial recovery for English-only models, indicating that code-switching failures stem from deeper representation and training mismatches, not just surface-level vocabulary.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.