Reproduce “Formalized Information Needs Improve Large-Language-Model Relevance Judgments” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer

Researchers from TH Köln, Friedrich-Schiller-Universität Jena, and Aalborg University demonstrated that providing Large Language Models (LLMs) with formalized information needs, similar to human assessors, improves the reliability and reusability of their relevance judgments in information retrieval evaluations, reducing positivity bias and increasing inter-assessor agreement.

Key insights

  • Formalized information needs, even when synthetically generated by LLMs, consistently improve the reliability and reduce the positivity bias of LLM relevance judgments compared to query-only inputs.
  • The `query-contrastive` prompt, which integrates queries with both relevant and non-relevant documents, proved most effective for synthesizing topics that enhance LLM judgment quality.
  • Applying formalized topics significantly boosts inter-LLM assessor agreement and the reusability of LLM-generated test collections, making system rankings more stable.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.