Reproduce “HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tingting Chen, Beibei Lin, Zifeng Yuan, Qiran Zou, Hongyu He, Anirudh Goyal, Yew-Soon Ong, Dianbo Liu

HypoSpace introduces a diagnostic benchmark to evaluate large language models (LLMs) in generating diverse sets of hypotheses under conditions of underdetermination. This framework reveals that current LLMs exhibit "mode collapse," generating limited subsets of valid explanations despite maintaining high individual hypothesis validity, particularly as the solution space increases in complexity.

Key insights

  • LLMs, even frontier models, consistently exhibit "mode collapse" when exploring underdetermined hypothesis spaces; they generate many valid hypotheses but repeatedly sample from a narrow subset, leading to low uniqueness and recovery.
  • Theoretical analysis demonstrates that if an LLM's internal distribution over hypotheses is highly 'peaked,' achieving comprehensive coverage requires an impractically large sampling budget, explaining empirical mode collapse.
  • Simple interventions like increased sampling budgets or changes to decoding parameters (temperature, top-p) are insufficient to mitigate mode collapse, indicating a structural bias in the models' sampling distributions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.