Reproduce “SLATE: A Large-Scale Benchmark for Long-Horizon API Selection in E-commerce Tool Use” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rongzhe Wei, Ge Shi, Min Cheng, Sarthak Ghosh, Vaibhav Gorde, Leman Akoglu, Pan Li, Na Zhang

SLATE benchmarks enterprise API selection by turning e-commerce requests into hierarchical procedures evaluated in a deterministic simulator. It contains 100 synthetic tasks and 1,000 tools. Models choose among 52 candidates, with reference arguments supplied for correctly selected tools. The best tested model achieves 5.0 ± 0.8% task success, a result limited to tool selection rather than live, autonomous execution.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.