Reproduce “AInsteinBench: Benchmarking Coding Agents on Scientific Repositories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Titouan Duston, Shuo Xin, Yang Sun, Daoguang Zan, Aoyan Li, Shulin Xin, Kai Shen, Yixiao Chen and 21 more
AInsteinBench introduces a large-scale benchmark comprising 244 problems from six real scientific software repositories to evaluate large language model (LLM) agents on scientific computing development tasks. Frontier models achieved resolved rates up to 44.0%, revealing challenges in domain-specific reasoning, cross-file coherence, and numerical stability unique to scientific codebases.
Key insights
- Scientific software development requires adherence to physical invariants, mathematical structure, and established community conventions, which current LLMs frequently fail to uphold.
- LLM agents exhibit varied performance across scientific domains; for example, they struggle with genuine spatial reasoning (RDKit) and high engineering complexity (AMReX) but perform better with tasks rich in tooling (Einstein Toolkit).
- Beyond generic software errors, models often produce 'locally plausible but scientifically invalid' solutions, failing due to subtle global inconsistencies, mathematical inaccuracies, or violations of conservation laws.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.