Reproduce “llmNER: (Zero|Few)-Shot Named Entity Recognition, Exploiting the Power of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabián Villena, Luis Miranda, Claudio Aracena
llmNER is a Python library designed to streamline zero-shot and few-shot Named Entity Recognition (NER) using Large Language Models (LLMs) by abstracting prompt engineering and response parsing. The library's evaluation on CoNLL 2003 (English) and CoNLL 2002 (Spanish) datasets showed GPT 3.5's superior performance, the general benefit of few-shot examples, and that an "in-line" answer format often yielded better results than JSON.
Key insights
- The choice of underlying LLM significantly impacts zero-shot/few-shot NER performance, with GPT 3.5 consistently outperforming other models like Mixtral 8x7B and Llama 2 70B.
- Incorporating a few examples into prompts generally enhances LLM performance for NER, validating in-context learning benefits.
- "In-line" answer shapes for NER outputs can be more effective than JSON formats, suggesting that preferred output structures for LLMs might differ from common expectations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.