Reproduce “ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
SHORTCUTSBENCH provides a large-scale, real-world benchmark for API-based LLM agents, constructed from Apple Shortcuts data. Its evaluations show that current agents struggle with multi-step reasoning for API selection, accurate parameter filling, and identifying when missing input is required for complex tasks, effectively differentiating model capabilities.
Key insights
- SHORTCUTSBENCH effectively distinguishes between LLM agents of varying intelligence, unlike prior benchmarks which often showed a "ceiling effect."
- Multi-step reasoning for API selection and accurate parameter extraction from diverse user queries are primary bottlenecks for current LLM agents.
- A widespread deficiency exists in agents' ability to recognize and request necessary missing input from the system or user for task completion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.