Reproduce “QMBench: A Research Level Benchmark for Quantum Materials Research” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanzhen Wang, Yiyang Jiang, Diana Golovanova, Kamal Das, Hyeonhu Bae, Yufei Zhao, Huu-Thong Le, Abhinava Chatterjee and 5 more
QMBench introduces a research-level benchmark containing 103 expert-curated problems to evaluate large language model (LLM) agents in quantum materials research, using diverse output types including atomic structure files (POSCAR). Evaluations of leading LLMs revealed strong knowledge recall but weaknesses in analytical reasoning, quantitative figure interpretation, and complex computational workflows.
Key insights
- Leading LLMs demonstrated proficiency in knowledge-oriented questions requiring encyclopedic recall of quantum materials concepts but exhibited systematic weaknesses in applied reasoning and practical execution.
- Models struggled particularly with rigorous analytical calculations, multi-step derivations, and quantitative interpretation of complex scientific figures.
- Generating and manipulating complex atomistic structural files, such as POSCARs for heterostructures or slabs, proved challenging for models beyond basic familiarity with the format.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.