Reproduce “SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou and 12 more
The authors introduce SCI-AGENT-EXAM, a benchmark of 120 executable tasks across 11 scientific domains and seven workflow types. It requires structured answers, executable artifacts, and reports, scoring execution separately from report quality. In tests of 10 agents, GPT-6 Astra scored 80.9/100, yet reached a near-perfect combined score in 26% of runs. The benchmark measures curated tasks, not broad scientific autonomy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.