Reproduce “Automatic In-Domain Exemplar Construction and LLM-Based Refinement of Multi-LLM Expansions for Query Expansion” on OpenResearch

About this paper

Minghan Li, Ercong Nie, Siqi Zhao, Tongna Chen, Huiping Huang, Guodong Zhou

An automated, domain-adaptive framework for query expansion was developed, integrating cluster-based in-domain exemplar construction with a multi-LLM ensemble and refinement module. This approach consistently enhanced retrieval effectiveness, showing gains such as a +4.15 absolute improvement in NDCG@10 on TREC DL20 compared to single-LLM methods.

Key insights

• Automating the construction of domain-adaptive and diverse in-domain exemplars is critical for achieving stable and effective performance in LLM-based query expansion.
• Leveraging the complementary knowledge from multiple heterogeneous LLMs through an LLM-based refinement step effectively consolidates and enriches query expansions, outperforming simple concatenation or single-model approaches.
• Controlled, quality-focused multi-LLM refinement that synthesizes information is more beneficial than merely increasing the maximum expansion length, as excessive verbosity from a single LLM can introduce noise and degrade retrieval effectiveness.

Summary by alphaXiv.