Reproduce “QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Belinda Z. Li, Been Kim, Zi Wang
QuestBench, a new benchmark, evaluates how well large language models (LLMs) can ask clarifying questions to acquire missing information for reasoning tasks. It reveals that while LLMs perform well in mathematical information acquisition, they struggle significantly with underspecified logic and planning problems, and their ability to solve a fully specified problem does not guarantee their capacity to ask the right clarifying questions.
Key insights
- There is a notable disparity in LLM performance for information acquisition, with high accuracy in mathematical reasoning (GSM-Q/GSME-Q) but significant struggles in logic (Logic-Q) and planning (Planning-Q) domains.
- The capacity to solve a problem when all information is provided (well-specified) does not guarantee the ability to identify and ask for missing information in an underspecified version.
- LLMs demonstrate limited "epistemic humility," often failing to detect underspecification and admit uncertainty, preferring to attempt a solution even when crucial information is absent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.