Reproduce “UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang and 1 more

UGPhysics, a large-scale benchmark comprising 5,520 undergraduate physics problems, was introduced alongside the Model-Assistant Rule-based Judgment (MARJ) framework to assess large language models' (LLMs) physics reasoning capabilities. Evaluations revealed that even top LLMs struggle significantly with undergraduate physics, with the highest accuracy achieved being 49.8%, and highlighted distinct challenges compared to pure mathematical reasoning.

Key insights

  • Physics reasoning presents a distinct and more substantial challenge for LLMs than pure mathematical reasoning, with even top models achieving significantly lower accuracy on UGPhysics compared to math benchmarks.
  • Math-specialized training yields only marginal performance improvements in physics reasoning, underscoring the necessity for dedicated, physics-specific content in future LLM training.
  • LLMs exhibit particular weakness in 'Math Derivation' tasks within a physics context, suggesting difficulties in integrating physical knowledge with complex mathematical operations and practical interpretations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.