Reproduce “Language Fairness in Multilingual Information Retrieval” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Eugene Yang, Thomas Jänich, James Mayfield, Dawn Lawrie

A new metric, PEER (Probability of Equal Expected Rank), quantifies language fairness in Multilingual Information Retrieval (MLIR) systems by assessing if documents of the same relevance level are presented at equivalent expected ranks across languages. Experiments on synthetic and real-world benchmarks show that different MLIR approaches, including neural models, exhibit measurable language preferences, which PEER effectively identifies.

Key insights

  • PEER uniquely defines language fairness as the "equal treatment of all groups," where no language is inherently disadvantaged, diverging from traditional "protected group" fairness paradigms.
  • The metric's non-parametric basis (Kruskal-Wallis test) ensures its robustness and applicability across various MLIR architectures without distributional assumptions about scores.
  • PEER provides insights into language balance independently of traditional retrieval effectiveness metrics, unlike some existing fairness measures that correlate strongly with recall.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.