Reproduce “BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fei Lin, Tengchao Zhang, Ziyang Gong, Xiaotong Yu, Bohan Zhang, Yifan Zhou, Qihao Yang, Ji Dai and 10 more
BEAKER introduces an expert-authored benchmark for testing whether models can judge actionable regions, feasible paths, and other execution-relevant details in laboratory images and videos. It evaluates 28 models across four capabilities using offline, single-turn questions. Actionability has the lowest mean score, 22.2, while Experimentation averages 58.0; the benchmark does not establish real-robot performance or deployment safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.