Reproduce “HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li, Xander Xu, Bohan Wang, Peng Wang and 28 more
HLE-Verified audits and repairs flawed items in the 2,500-item HLE benchmark. Experts assess each problem, answer, and rationale, use model solutions as supporting evidence, and leave unresolved cases marked uncertain. In text-only evaluations, GPT-5.2-High gains 9.95 percentage points on the full set and 38.04 on items whose problem or answer was revised; 689 items remain uncertain.
Key insights
- The reliability degradation in the original HLE benchmark is primarily driven by "Incorrect Answer" at the answer level and by information gaps or format errors at the rationale and problem levels.
- Benchmark flaws vary significantly across subject categories; for example, formal scientific domains show high problem validity but frequent rationale errors, while other domains suffer from high uncertainty.
- Model self-reported confidence can serve as a diagnostic signal for identifying noisy or ambiguous items, as confidence consistently increased after problem-level repairs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.