Reproduce “AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu and 11 more

AlphaDiana evaluates reasoning agents by measuring models and harnesses under shared task and resource policies while preserving each harness’s control loop. It links outcomes to execution traces and resource records. In Qwen3.5-27B on GPQA-Diamond, a combined set of Direct and three harnesses solved 175 of 198 questions, versus 159 for Direct; this coverage does not test system selection.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.