Reproduce “mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Orion Weller, Benjamin Chang, Eugene Yang, Mahsa Yarmohammadi, Sam Barham, Sean MacAvaney, Arman Cohan, Luca Soldaini and 2 more
mFollowIR, a collaborative effort from Johns Hopkins University and others, introduces the first multilingual benchmark for evaluating how retrieval systems follow natural language instructions across Russian, Chinese, and Persian. The benchmark reveals that while instruction training improves performance, current models struggle with non-English instructions, often failing to correctly re-rank documents based on nuanced changes despite maintaining general retrieval effectiveness.
Key insights
- Models explicitly trained with instructions exhibit superior instruction-following performance, particularly in cross-lingual settings, with Promptriever-Llama3.1-Instruct achieving the highest p-MRR of 10.4.
- A substantial performance drop occurs when models move from cross-lingual (English instructions) to multilingual (non-English instructions) scenarios, indicating an English-centric bias in current instruction-following capabilities.
- Many models can achieve high general retrieval effectiveness (nDCG@20) while simultaneously failing to follow nuanced instructions (low or negative p-MRR), highlighting a disconnect between semantic matching and precise instruction adherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.