Reproduce “PARLEY: Evaluating LLM Collaboration Through Cooperative Asymmetric Puzzles” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Javier Hernandez, Andrew D. Wilson

PARLEY tests whether agents with unequal information and tools can coordinate on shared tasks. Two GPT-4o agents communicate in natural language across four controlled puzzles, while the environment verifies completion; the authors analyze failure patterns and add six repair instructions to both prompts. Completion rose from 60.5% to 70.5%, but the study covers only GPT-4o and these tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.