Reproduce “MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu and 5 more

MerchantBench, a new benchmark for LLM agents, evaluates their "Long-Term Coherence" in simulated seller-side e-commerce operations that feature mixed-latency feedback and nonstationary demand. The benchmark reveals that current LLM agents achieve approximately 27.3% of human performance in maximizing net assets over a 365-day simulation, highlighting significant gaps in sustained, adaptive decision-making.

Key insights

  • Current LLM agents demonstrate a substantial performance gap compared to human operators in sustained e-commerce operations, achieving only a fraction of human-level net assets.
  • Observed failure modes include "Operational Coherence" issues, such as activity decay and reduced tool engagement over time, and "Strategic Coherence" failures like control-loop narrowing and premature abandonment.
  • LLM agents struggle with "Evidence-Calibrated Adaptation," exhibiting difficulty in effectively revising policies in response to evolving liquidity, seasonal demand, and heterogeneous, delayed feedback.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.