Reproduce “Knowing What's Missing: Assessing Information Sufficiency in Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Akriti Jain, Aparna Garimella
Jain and Garimella from Adobe Research developed an "Identify-then-Verify" framework to robustly assess information sufficiency in Large Language Model (LLM) based question answering, particularly for inferential questions. This method reformulates sufficiency as explicitly identifying what specific information is missing, resulting in improved accuracy on complex benchmarks and providing actionable insights into information gaps.
Key insights
- Information sufficiency exists on a spectrum from pragmatic inference to strict literal matching, and the framework can adapt to different requirements by tuning the verification strictness.
- Explicitly identifying *what* specific information is missing (rather than a binary judgment) compels the model to form concrete hypotheses and yields granular, actionable output beneficial for downstream applications.
- Leveraging self-consistency across multiple LLM generations combined with semantic consensus and an attribution verification step significantly mitigates the instability and hallucination risks associated with single LLM judgments on information gaps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.