Reproduce “Improving the Reusability of Conversational Search Test Collections” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, Mohammad Aliannejadi
A research team from the University of Amsterdam and University of Strathclyde develops methods to improve conversational search test collections by using large language models (LLMs) to fill missing relevance judgments, demonstrating that fine-tuned Llama-3.1 achieves high agreement with human assessors while maintaining ranking correlation across evaluation scenarios.
Key insights
- Fine-tuned Llama-3.1 shows higher agreement with human assessors compared to prompt-based approaches
- Zero-shot Llama-3.1 causes fewer ranking shifts than one-shot GPT-3.5 when filling judgment holes
- For systems with many missing judgments, recreating the whole pool with one-shot GPT-3.5 is better than partial augmentation
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.