Reproduce “Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nandan Thakur, Crystina Zhang, Xueguang Ma, Jimmy Lin
An LLM-cascading framework, RLHN, was developed to identify and relabel false negatives within large-scale information retrieval training datasets like MS MARCO. This data curation method enhances the robustness and generalization of retrieval and reranker models, leading to average nDCG@10 improvements of 0.7 to 1.4 points on the BEIR benchmark and stronger performance on out-of-domain tasks.
Key insights
- Simply increasing training data volume without addressing quality issues, particularly false negatives, can degrade retrieval model performance; pruning certain datasets can paradoxically improve it.
- False negatives are widespread in existing large-scale, sparsely-annotated IR datasets, with up to 56% of MS MARCO training pairs containing at least one.
- A cascading LLM approach offers a cost-effective and reliable method for identifying and relabeling these false negatives at scale, validated by human agreement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.