Reproduce “LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling” on OpenResearch

About this paper

Liam Johnston, Shayan Noei, Maram Assi, Ying Zou

LabelMate, an LLM-driven framework developed by researchers from Queen's University, Université du Québec à Montréal, and other institutions, automates the derivation of project-specific label taxonomies and the assignment of labels to software issue reports. Leveraging a retrieval-augmented generation (RAG) mechanism, the framework achieved an 89.84% overall label accuracy and 81.34% of issue reports with 100% label accuracy, demonstrating statistically significant improvements in accuracy and efficiency over baseline methods without requiring pre-labeled training data.

Key insights

• Unconstrained LLM label generation results in highly fragmented label spaces with thousands of unique, often over-specific or synonymous labels, necessitating a structured consolidation approach.
• Original developer-assigned labels may not always serve as an ideal ground truth, as LLM-assigned labels can offer more specific technical descriptions for issue reports.
• Integrating a RAG mechanism significantly improves label assignment accuracy and efficiency, with semantic alignment (cosine similarity) serving as a strong proxy for label quality and demonstrating robustness even when retrieving issues from different domains.

Summary by alphaXiv.