Reproduce “AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taoyuze Lv, Alexander Chen, Fengyu Xie, Chu Wu, Jeffrey Meng, Dongzhan Zhou, Yingheng Wang, Bram Hoex and 2 more
AtomWorld evaluates whether large language models can execute precise atomic-structure modifications from CIF files and natural-language instructions, rather than merely describe or generate materials. Across 2,500 tasks, models handled simple composition changes relatively well but struggled with rotations, atom-count growth, long structures, and property-directed edits; generating executable pymatgen code improved performance, although rotation remained difficult.
Key insights
- Models are comparatively capable at changing, removing, swapping, or adding atoms, but multi-atom nonlinear transformations—especially rotation around a center—remain substantially harder even when CIF syntax is removed.
- Increasing structure length is a major failure source: in controlled insertion tests, performance declined from 12 to 216 atoms and all tested models failed beyond roughly 200 atoms; target atoms near the end of a CIF were also harder to manipulate.
- Tool use improves simple and intermediate edits markedly, but only raises rotation success from 6.8% to 18% for DeepSeek Chat, indicating that the main bottleneck is not formatting alone but reliable geometric execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.