Reproduce “PhysicsEval: Inference-Time Techniques to Improve the Reasoning Proficiency of Large Language Models on Physics Problems” on OpenResearch

About this paper

Oshayer Siddique, J. M Areeb Uzair Alam, Md Jobayer Rahman Rafy, Syed Rifat Raiyan, Hasan Mahmud, Md Kamrul Hasan

Researchers from the Islamic University of Technology introduced PHYSICSEVAL, a large-scale benchmark containing 19,609 physics problems, and evaluated inference-time techniques to enhance Large Language Models' (LLMs) physics reasoning. A multi-agent review framework consistently improved LLM performance, achieving average Physics Proficiency Scores of 91.78 on easy and 68.23 on hard problems, while self-refinement sometimes degraded results for certain models.

Key insights

• External critique, particularly from a multi-agent framework, consistently leads to greater improvements in LLM physics reasoning proficiency, with gains becoming more pronounced on difficult problems.
• Self-refinement, where an LLM attempts to correct its own solution without external guidance, can surprisingly degrade performance for some models, suggesting limitations in internal metacognitive processes.
• The multi-agent framework offers a cost-effective strategy by leveraging smaller, open-source LLMs as verifiers to scrutinize solutions from more powerful, commercial LLMs.

Summary by alphaXiv.