Reproduce “PhySense: Principle-Based Physics Reasoning Benchmarking for Large Language Models” on OpenResearch

About this paper

Yinggan Xu, Yue Liu, Zhiqiang Gao, Changnan Peng, Di Luo

Researchers from UCLA, Caltech, UC Berkeley, and MIT developed PhySense, a specialized benchmark to assess Large Language Models' ability in principle-based physics reasoning. The study revealed that current LLMs struggle significantly with applying fundamental physical principles, often requiring 10 to 100 times more computational steps than human experts and showing only marginal improvements from explicit guidance.

Key insights

• Large language models consistently underperformed on principle-based reasoning tasks, demonstrating fundamental challenges in applying core physical principles.
• A striking efficiency gap exists, with LLMs using 10 to 100 times more tokens (1,000-10,000) than human experts (around 100) to solve problems, highlighting a reliance on computationally intensive methods over elegant principle-driven approaches.
• Prompting strategies, including providing hints or instructing to avoid complex computations, yielded only marginal improvements, suggesting a deeper issue in how LLMs apply principles rather than just recognizing them.

Summary by alphaXiv.