Reproduce “A Sequential Decision Framework for Evaluating LLM-Guided Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siqing Huo, Andrew Parry, Debasis Ganguly, Mark Smucker, Charles L. A. Clarke
The authors present a framework for evaluating search policies that allocate effort across ranked document lists. It represents continuing in an existing list or generating a new one, using relevance judgments to guide later choices. On TREC Deep Learning 2019, SPLADE with query reformulation and MTF reports average recall@R of 0.531 versus 0.414 for ad hoc search; the intervals overlap.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.