Reproduce “ExpMRC: Explainability Evaluation for Machine Reading Comprehension” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Cui, Ting Liu, Wanxiang Che, Zhigang Chen, Shijin Wang
A new benchmark, ExpMRC, is introduced for evaluating the explainability of Machine Reading Comprehension (MRC) systems across English and Chinese, and both span-extraction and multi-choice tasks. The benchmark, which focuses on unsupervised evidence extraction, reveals that current state-of-the-art models are still significantly below human performance in generating explanations.
Key insights
- Current state-of-the-art MRC models are significantly behind human performance in providing accurate explanations, especially for multi-choice tasks.
- The effectiveness of evidence extraction methods varies by task type, with token-similarity approaches performing better for span-extraction and pseudo-data training being more suitable for multi-choice tasks.
- Integrating question context is crucial for identifying relevant evidence, and there is substantial room for improvement in extracting precise, sub-sentence evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.