Reproduce “Benchmarking language models and agents for scientific reasoning in evolving materials laboratories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuyu Dong, Yixuan Li, Jianguo Huang, Feng Chen, Yiyun Zhou, Bo An, Zheng Liu, Da Shu
MatScope benchmarks whether models can make condition-specific materials judgments and update them as evidence arrives. It tests unaided predictions, literature-assisted transfer, and forecasts revised while targets remain hidden. Across eleven deployments, given-input calculations score 92.38–99.05%, versus 22.74–31.21% for conditional predictions. In ME20, Fable’s score rises from 21.42% to 37.67%. The benchmark focuses on alloys, not live laboratory readiness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.