Reproduce “Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents” on OpenResearch

About this paper

Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan and 1 more

Collab-Overcooked introduces a rigorous benchmark and evaluation framework for Large Language Model-based Multi-Agent Systems, strictly enforcing collaboration in an Overcooked-AI environment with 30 tasks across 6 complexity levels. The framework reveals that LLMs struggle with proactively initiating collaboration and exhibit attention biases that hinder sustained teamwork, with human performance far exceeding even top models in consistency and adaptive reasoning.

Key insights

• LLMs exhibit an imbalance in collaborative skills, being generally better at responding to explicit requests than proactively identifying needs and initiating collaborative actions.
• Current LLM-MAS struggle with sustained, long-term coherence and adaptation in complex collaborative tasks, showing performance degradation and a "positional dependence" where identical actions fail more often later in a sequence.
• Collaboration failures are causally linked to attention biases where LLMs misprioritize information, such as overemphasizing task execution details (recipe) while undervaluing essential collaboration rules.

Summary by alphaXiv.