Reproduce “SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jinwoo Hwang, Sukmin Cho, Tao Zhang, Soheil Abbasloo, Jongse Park, Peng Cheng

SIRIS tests whether language-model agents can judge when a simulator flaw changes the best systems design. It pairs workloads where the same perturbation affects predictions, but the reference simulator overturns the recommendation in only one. GPT-6 Astra achieved 95% judgment, solving both cases in 79 of 87 pairs. These choices are reference-defined, not evidence of real-hardware performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.