Reproduce “Comparative Analysis of Encoder-Based NER and Large Language Models for Skill Extraction from Russian Job Vacancies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikita Matkin, Aleksei Smirnov, Mikhail Usanin, Egor Ivanov, Kirill Sobyanin, Sofiia Paklina, Petr Parshakov
Researchers from HSE University conducted a comparative analysis of encoder-based Named Entity Recognition (NER) models and Large Language Models (LLMs) for skill extraction from Russian job vacancies. The study determined that fine-tuned encoder-based NER models delivered comparable accuracy to fine-tuned LLMs (F1 score of 0.81) but with significantly faster inference times and lower operational costs.
Key insights
- Fine-tuning significantly enhances LLM performance for skill extraction, allowing some fine-tuned LLMs to match the F1 scores of specialized encoder-based NER models.
- Encoder-based NER models, despite being smaller, demonstrated superior efficiency and cost-effectiveness for structured skill extraction tasks in Russian compared to both off-the-shelf and fine-tuned LLMs.
- Robust semantic similarity metrics are critical for fairly evaluating and comparing the diverse output formats of generative LLMs and token-level extractive NER models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.