Reproduce “Extreme Multi-Label Skill Extraction Training using Large Language Models” on OpenResearch

About this paper

Jens-Joris Decorte, Severine Verlinden, Jeroen Van Hautte, Johannes Deleu, Chris Develder, Thomas Demeester

Researchers from Ghent University – imec and TechWolf developed a methodology to train extreme multi-label skill extraction models by generating high-quality synthetic training data using Large Language Models (LLMs). This approach, leveraging an enhanced prompt design and a contrastive learning bi-encoder, achieved R-Precision@5 improvements of 15 to 25 percentage points over distant supervision baselines on multiple benchmarks.

Key insights

• Refined LLM prompt design, incorporating skill descriptions and specifying hypothetical scenarios, significantly improved synthetic data coverage (from 73% to 99.5%) and quality (from 88% to 94% accuracy).
• Using LLMs for synthetic data generation at training time is more efficient and effective than using them for inference, circumventing issues of non-determinism, cost, and latency.
• Training with synthetically generated job ad-like sentences provides a more relevant and superior signal for skill extraction compared to using generic ESCO skill descriptions, even with minimal data quantities.

Summary by alphaXiv.