Reproduce “Perspectives on Large Language Models for Relevance Judgment” on OpenResearch

About this paper

Guglielmo Faggioli, Laura Dietz, Charles Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas and 3 more

Large Language Models (LLMs) are examined for their potential in information retrieval relevance judgment, proposing a human-machine collaboration spectrum and empirically comparing LLM-generated judgments against human assessments on TREC datasets. The analysis indicates LLMs can substantially lower judgment costs but exhibit lower agreement with human judgments and reduced discriminative power for distinguishing retrieval system performance.

Key insights

• A "Human–Machine Collaboration Spectrum" offers a structured way to consider various integration points for LLMs in relevance assessment, from human-centric to fully automated roles.
• While LLMs can drastically reduce the cost of obtaining judgments, their agreement with human judgments and their ability to discriminate between retrieval system performances are lower than human assessors.
• LLMs often over-predict higher relevance levels and under-predict non-relevant documents when compared to human assessments, suggesting a bias in their judgment distribution.

Summary by alphaXiv.