Reproduce “How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi and 43 more

The authors audit six physics benchmarks by having experts check questions, reference answers, and model responses, allowing equivalent mathematical forms and repairing or excluding flawed items. In one example, a grader rejects an expression algebraically identical to the reference. GPT-5.6-Sol High’s HLE-Physics mean@4 rises from 47.28% to 78.66% on a different set; this concerns closed-ended problems, not physics research overall.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.