Reproduce “EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiqing Cui, Xinxiang Yin, Yihong Tang, Xinglang Zhang, Yuanzhe Hu, Siru Zhong, Weidong Tang, Yuxuan Liang and 6 more

EarthVerse, a benchmark from a multi-institutional team including MIT and HKUST(GZ), evaluates scientific agents' ability to maintain the correspondence between heterogeneous observations and scientific claims across dynamic Earth systems and natural hazards. It revealed a "reliability gap," where current agents achieve high accuracy on individual steps but fail to maintain consistent, auditable evidence chains, with the best performing agent achieving 84.65% mean answer-unit accuracy but only 34.81% Strict@95.

Key insights

  • A "reliability gap" exists where agents achieve high accuracy on individual analytical components but fail to maintain an auditable, consistent chain across evidence, scales, units, calculations, and physical interpretation in a majority of investigations.
  • Reasoning improves scientific investigations primarily when the system can dynamically edit and revise its claim–evidence state, with increased effort correlating to higher scores in interactive settings where evidence can be retrieved or findings revised.
  • Evidence localization (autonomously finding and aligning relevant observations) is the most impactful intervention and a critical bottleneck, enhancing performance in spatiotemporal reconstruction, physical-mechanism reasoning, and quantitative calculation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.