Reproduce “Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Axel Backlund, Lukas Petersson

Andon Labs developed Vending-Bench, a new benchmark designed to assess the long-term coherence of LLM-based agents within a simulated vending machine business environment. The research found that while some LLMs can achieve high peak performance, they consistently exhibit significant variance across runs and struggle to maintain sustained, coherent reasoning over extended periods.

Key insights

  • The quality of memory management appears more critical than raw memory capacity for long-term coherence, with models performing better with less token memory in some instances.
  • LLM agents exhibit high variance in performance across multiple runs, indicating a fundamental instability in their long-term coherent behavior.
  • Common failure modes include misinterpreting delivery schedules, forgetting past orders, and entering repetitive, ineffective 'meltdown' loops.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.