Reproduce “Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bryan Li, Fiona Luo, Samar Haider, Adwait Agashe, Tammy Li, Runqi Liu, Muqing Miao, Shriya Ramakrishnan and 2 more
The paper introduces BORDIRLINES, a benchmark for evaluating multilingual Retrieval-Augmented Generation (RAG) in culturally-sensitive contexts like territorial disputes. It demonstrates that retrieving documents from diverse languages generally enhances factual accuracy and reduces geopolitical bias in LLM responses, often improving cross-lingual consistency and even neutralizing bias for some models.
Key insights
- RAG consistently improves factual accuracy and reduces geopolitical bias in LLM responses for culturally sensitive tasks, but its impact on cross-lingual consistency varies based on the RAG mode and the LLM used.
- Retrieving documents from diverse languages (`rel_langs` mode) proves more effective at reducing geopolitical bias and often enhances cross-lingual consistency compared to relying solely on in-language documents.
- LLMs exhibit varied citation behaviors across languages, with low-resource languages showing higher variability and an in-language preference in citation rates, highlighting potential inequities in multilingual RAG.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.