Reproduce “AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu and 11 more
AlphaDiana evaluates reasoning agents by measuring models and harnesses under shared task and resource policies while preserving each harness’s control loop. It links outcomes to execution traces and resource records. In Qwen3.5-27B on GPQA-Diamond, a combined set of Direct and three harnesses solved 175 of 198 questions, versus 159 for Direct; this coverage does not test system selection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.