Reproduce “Scaling Physical Reasoning with the PHYSICS Dataset” on OpenResearch

About this paper

Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu and 4 more

The research introduces the PHYSICS dataset, a large-scale collection of 16,568 bilingual physics questions, and a Rule+Model hybrid evaluation framework to address limitations in current LLMs' physical reasoning. This work provides a more accurate assessment tool and demonstrates that fine-tuning models with PHYSICS leads to improved performance on physics, mathematical, and general reasoning benchmarks.

Key insights

• Strong mathematical reasoning in LLMs does not directly translate to equally strong complex physical reasoning, with top-performing models achieving around 50-60% accuracy on physics benchmarks.
• The Rule+Model hybrid evaluation framework significantly enhances evaluation accuracy for physics problems (over 20% improvement) compared to purely rule-based methods, addressing physics-specific nuances.
• Supervised fine-tuning with the PHYSICS dataset leads to notable performance improvements across various physics, mathematical, and general reasoning benchmarks, indicating a positive transfer of learned skills.
• Error analysis reveals that knowledge deficits, particularly conceptual and modeling errors, constitute the majority of LLM failures in physics, highlighting the need for deeper domain-specific understanding.

Summary by alphaXiv.