Reproduce “Solving Physics Olympiad via Reinforcement Learning on Physics Simulators” on OpenResearch

About this paper

Mihir Prabhudesai, Aryan Satpathy, Yangmin Li, Zheyang Qin, Nikash Bhardwaj, Amir Zadeh, Chuan Li, Katerina Fragkiadaki and 1 more

Carnegie Mellon University and Lambda researchers developed Sim2Reason, a framework that leverages physics simulators to generate scalable, high-fidelity question-answer pairs for training large language models in physical reasoning. This approach enables LLMs to achieve zero-shot sim-to-real transfer and improve performance on real-world physics benchmarks, including up to a +17.9% gain on JEEBench and competitive accuracy on IPhO mechanics.

Key insights

• Physics simulators, when paired with a structured data generation and filtration pipeline, can serve as a scalable source of high-fidelity, verifiable supervision for training LLMs in complex physical reasoning.
• Reinforcement Learning with Verifiable Rewards (RLVR) on synthetic data enables zero-shot sim-to-real transfer, allowing LLMs to generalize physical reasoning skills to real-world problems like the International Physics Olympiad.
• Explicitly filtering out "shortcut solutions" from the generated data, which do not require deep multi-entity reasoning, is essential for improving the quality of LLM training and enhancing out-of-distribution generalization.

Summary by alphaXiv.