Reproduce “When the World Is the Test Suite: Agentic Intelligence for the Physical World” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaixin Li, Meng Luo, Yanlin Li, Yuchen Tian, Xiang Liu, Xiangru Jian, Yan Yang, Zhiheng Xi and 4 more
The authors survey how programs for physical systems are tested, repaired, and retained, distinguishing program roles and examining traces, success judges, simulation, and formal checks. They report that the best of thirteen vision-language-model detectors on FailBench achieved 0.77 macro-averaged balanced accuracy, not task-success rate, and argue that physical-world tests and their acceptance criteria also require validation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.