Reproduce “CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese” on OpenResearch

About this paper

Liang Xu, Yu tong, Qianqian Dong, Yixuan Liao, Cong Yu, Yin Tian, Weitang Liu, Lu Li and 2 more

Researchers from the CLUE Organization introduce CLUENER2020, a new fine-grained Chinese Named Entity Recognition dataset featuring 10 distinct entity categories and 13,436 annotated samples. This dataset provides a more challenging benchmark for Chinese NER, with state-of-the-art RoBERTa-NER achieving an F1 score of 80.42%, considerably lower than results on prior datasets.

Key insights

• CLUENER2020 presents a significantly more challenging benchmark for Chinese NER, with the best baseline model achieving an F1 score of 80.42%, indicating substantial room for algorithmic improvement.
• Pre-trained language models like BERT-NER and RoBERTa-NER consistently outperformed traditional architectures such as BILSTM-CRF-NER, with RoBERTa-NER demonstrating superior performance.
• Human annotators achieved a lower overall F1 score (63.41%) compared to advanced machine learning models on this fine-grained task, which suggests that models can surpass human performance in complex, precisely defined NER scenarios given ample training data.

Summary by alphaXiv.