Reproduce “When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
William A. Ingram, Bipasha Banerjee, Edward A. Fox
Research from Virginia Tech characterizes how large language models exhibit systematic relevance filtering biases and retrieval divergence when classifying documents for Sustainable Development Goals. It demonstrates that observed disagreements between models are lexically distinguishable and significantly alter the composition of top-ranked search results.
Key insights
- LLM disagreement in relevance labeling is systematic and reflects distinct, model-specific filtering preferences, rather than being random noise.
- These systematic filtering preferences are demonstrably linked to distinguishable lexical and thematic patterns within the documents being classified.
- Initial relevance label disagreements between LLMs profoundly impact downstream ranked retrieval, causing significant divergence in the composition of top-ranked documents even under shared scoring functions.
- The filtering behavior of LLMs is learnable from lexical content, suggesting that these biases are structured and predictable.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.