Reproduce “Dataset for the First Evaluation on Chinese Machine Reading Comprehension” on OpenResearch

About this paper

Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, Guoping Hu

Researchers from iFLYTEK Research and Harbin Institute of Technology created a novel Chinese Machine Reading Comprehension (MRC) dataset and organized the CMRC-2017 evaluation, fostering progress in Chinese NLP. The platform demonstrated that top ensemble systems achieved up to 81.90% accuracy on cloze tasks and 69.53% on the more challenging user query track.

Key insights

• A hybrid approach combining large-scale automatically generated data for training with carefully human-annotated data for validation and testing is crucial for creating robust and realistic MRC benchmarks.
• The significant performance drop observed in the User Query Track confirmed that a mismatch between training data style (cloze) and evaluation data style (natural queries) presents a substantial challenge for current MRC models.
• Ensemble methods prove highly effective in pushing the performance limits of neural models in competitive MRC tasks, achieving state-of-the-art results on the dataset.

Summary by alphaXiv.