Reproduce “I-PHYRE: Interactive Physical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiqian Li, Kewen Wu, Chi Zhang, Yixin Zhu
Researchers from Peking University, BIGAI, and Tsinghua University introduce I-PHYRE, a 2D physics benchmark designed to evaluate AI agents' interactive physical reasoning capabilities, requiring intuitive, multi-step, and in-situ interventions. Experiments reveal a substantial gap between current AI performance and human ability, with AI agents struggling particularly with generalization to complex compositional and multi-ball tasks and frequently failing due to imprecise action timing.
Key insights
- A significant performance disparity exists between human players and current AI agents in interactive physical reasoning, especially concerning generalization to complex scenarios.
- AI models struggle with zero-shot generalization to compositional and multi-ball tasks, which demand sophisticated multi-step planning and precise temporal control.
- A considerable portion of AI agent failures stems from incorrect action timing rather than merely suboptimal action order, highlighting the difficulty of in-situ interventions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.