Reproduce “Corpus-Steered Query Expansion with Large Language Models” on OpenResearch

About this paper

Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, Andrew Yates

Corpus-Steered Query Expansion (CSQE) leverages Large Language Models to extract relevant information directly from the retrieval corpus, combining it with LLM-generated expansions to significantly improve information retrieval performance and mitigate hallucinations across diverse datasets, often surpassing supervised methods without fine-tuning.

Key insights

• Leveraging Large Language Models (LLMs) for their relevance assessment capabilities to extract factual text from a corpus is more reliable for query expansion than relying solely on their generative knowledge.
• A hybrid approach that combines corpus-grounded expansions with LLM-generated hypothetical documents offers a robust balance of broad semantic understanding and factual accuracy.
• Intelligent query expansion can enable simple sparse retrievers to achieve performance competitive with or superior to heavily fine-tuned dense retrieval models, even in zero-shot settings.

Summary by alphaXiv.