Reproduce “FactoryBench: Evaluating Industrial Machine Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanis Merzouki, Coral Izquierdo, Matei Ignuta-Ciuncanu, Marcos Gomez-Bracamonte, Riccardo Maggioni, Alessandro Lombardi, Camilla Mazzoleni, Federico Martelli and 3 more
FactoryBench evaluates whether models can interpret industrial robot telemetry, predict intervention effects, reason about counterfactuals, and recommend protocol-grounded actions. It combines physical-robot episodes with structured questions and a free-form decision task. In zero-shot tests of six LLMs, none exceeded 50% chance-corrected score on Levels 1–3, while GPT-5.1 led Level 4 at 21.5%. This shows a ranking shift, not deployment readiness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.