Reproduce “TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
A. Ramesh Chandra, Aaghaz Mahajan, Ainesh Sanyal, Mianqi Wang, Shubhadeep Chakraborty, Sudip Chakraborty, Shailesh Chandrasekharan, Rama Govindarajan and 27 more
TheoPhysEval evaluates whether tool-using agents can complete answer-checkable theoretical-physics calculations using established methods. Its 50 problems cover 11 themes, and agents may use local computational tools, with or without web search. GPT-6-Astra with Codex passed 49 of 50 problems in the realistic track, showing high accuracy on these calculations, not breakthrough discovery or autonomous theory-building.
Key insights
- Leading agents handled many technically demanding calculations, but results reflect the ability to execute specified tasks—not to originate theories, select research directions, or judge a result’s importance.
- Performance changed only modestly between internet-enabled and isolated evaluations; the authors suggest models may retain substantial physics-literature knowledge, though single runs limit conclusions about the effect of web access.
- Failures included setup and formula-selection errors, while one ambiguous inverse problem defeated every evaluated model; some apparent failures also traced to errors in reference solutions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.