Reproduce “InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bryan L. M. de Oliveira, Luana G. B. Martins, Bruno Brandão, Luckeciano C. Melo

InfoQuest introduces a benchmark for testing whether assistants can uncover users’ unstated needs through conversation. It uses simulated users, scenario-specific checklist questions, and an automated judge to score checklist progress across up to 10 turns. In evaluation, Claude 3.7 Sonnet averaged 4.39 out of 5 and 7.99 turns; results are limited to this benchmark and setup.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.