Reproduce “EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyao Du, Tong Li, Haibo Zhang
EComAgentBench evaluates shopping agents on 662 long-horizon tasks in which customer requirements are distributed across the initial query, a tool-accessible profile, and conditional clarification responses. Using typed, source-labeled rubrics over 3.72 million products and 21.35 million reviews, it shows that hidden and implicit constraints—especially review-based requirements—remain difficult even for strong language models, with task accuracy ranging from 19.5% to 57.1%.
Key insights
- Hidden requirements were substantially harder than query-visible ones: for GPT-5.4, satisfaction was 88.1% for query rubrics but only 69.8% for persona rubrics and 70.9% for clarification rubrics.
- Implicit language drove difficulty more strongly than the total number of requirements: average accuracy fell from 51.8% with at most one implicit requirement to 27.2% with three or more.
- Review-opinion rubrics were the weakest category, largely because agents struggled to formulate effective searches and collect supporting review evidence before recommending a product.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.