Reproduce “MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junkai Zhang, Jingru Gan, Xiaoxuan Wang, Zian Jia, Changquan Gu, Jianpeng Chen, Yanqiao Zhu, Mingyu Derek Ma and 3 more
Researchers developed MatSciBench, a comprehensive and multimodal benchmark with 1,340 college-level materials science questions, to evaluate the reasoning capabilities of large language models. While top models like DeepSeek-R1 reached 75.22% accuracy on text-only problems, performance significantly declined to 53.02% on multimodal questions, primarily due to difficulties in visual data extraction.
Key insights
- Tool augmentation consistently enhances the performance of non-thinking LLMs on materials science tasks and offers a token-efficient approach to improve accuracy.
- Self-correction methods generally lead to performance degradation in LLMs, often by introducing incorrect revisions due to unreliable internal self-evaluation.
- Extracting precise numerical or quantitative information from scientific figures and plots represents a major bottleneck for multimodal LLMs in materials science.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.