Reproduce “CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, Takashi Ishida
CoffeeBench provides a new benchmark to evaluate large language model (LLM) agents in a 90-day simulated multi-agent economy featuring heterogeneous firms in a coffee supply chain. The evaluation demonstrates that top-performing models, such as GPT-5.5, can achieve a mean net income of "$+3,109" by actively engaging in communication and transactions, significantly outperforming rule-based baselines.
Key insights
- Active engagement in transactions and consistent inter-firm communication are crucial drivers for achieving higher profitability in complex multi-agent economic simulations.
- LLM agents can exhibit specific failure modes, such as "idle-drift," where they cease activity despite continuous internal reasoning, hindering long-term performance.
- A substantial performance gap exists between current top-performing LLM agents and theoretical optimal strategies, indicating considerable room for advancements in areas like price coordination and procurement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.