Reproduce “NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging” on OpenResearch

About this paper

Zihan Liu, Feijun Jiang, Yuxiang Hu, Chen Shi, Pascale Fung

NER-BERT, a model pre-trained on a newly constructed large-scale Named Entity Recognition corpus featuring 315 diverse entity types, addresses the challenge of low-resource entity tagging. The model achieves an average F1-score improvement of 4.87% over BERT in target-only fine-tuning settings across multiple domains, showing enhanced adaptation capabilities in data-scarce environments.

Key insights

• Task-specific pre-training on a large, high-quality NER corpus with diverse entity categories is more effective for low-resource NER than general PLMs or domain-adaptive pre-training.
• The granularity and abundance of entity types in the pre-training corpus significantly impact performance, with a richer set of fine-grained categories leading to better entity knowledge extraction.
• NER-BERT's task-specific pre-training provides robust pre-learned knowledge that allows for rapid adaptation and superior performance even in extremely low-resource (few-shot) settings.

Summary by alphaXiv.