Reproduce “XOR QA: Cross-lingual Open-Retrieval Question Answering” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, Hannaneh Hajishirzi

Researchers developed the XOR QA framework and the XOR-TYDI QA dataset to facilitate cross-lingual open-retrieval question answering, enabling questions in diverse non-English languages to be answered using content from other languages, particularly English. The research reveals that current state-of-the-art methods face considerable challenges in this cross-lingual setting, achieving significantly lower performance compared to monolingual QA.

Key insights

  • Machine translation performance significantly impacts downstream cross-lingual QA, with human-translated queries yielding substantially higher retrieval and QA scores compared to machine-translated ones.
  • Standard machine translation evaluation metrics, such as BLEU scores, do not consistently correlate with the practical performance of models in cross-lingual question answering.
  • Utilizing both English and native language Wikipedia collections for retrieval is generally more effective than relying on a single language, underscoring the benefits of a true cross-lingual search strategy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.