Reproduce “SemEval-2025 Task 5: LLMs4Subjects -- LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog” on OpenResearch

About this paper

Jennifer D'Souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, Diana Slawig

SemEval-2025 Task 5: LLMs4Subjects evaluated Large Language Models for automated subject classification within a national technical library's open-access catalog. The task established benchmarks for applying LLMs to GND-based subject indexing in a bilingual context, demonstrating that well-engineered LLM systems can predict over 50% of subject annotations and outperform general-purpose LLMs.

Key insights

• Multilingual models and diverse, augmented training data (including synthetic data generation and RAG) are crucial for enhancing LLM performance in complex, bilingual taxonomic classification tasks.
• Smaller, well-engineered LLM systems can achieve results comparable to or even surpass those from very large, instruction-tuned LLMs, emphasizing the importance of specialized system design and computational efficiency.
• General-purpose LLMs, even with prompt engineering and vector retrieval, exhibited reliability issues and ranked below specialized participant submissions, highlighting the need for tailored approaches in digital library contexts.

Summary by alphaXiv.