Reproduce “Scaling Discovery through Test-Time Communication” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos
The paper proposes verified progress sharing, a protocol in which long-running agents share experiments, artifacts, and measured improvements through a common workspace. Agents use feedback to adopt better approaches while retaining variation. On ARC-AGI-3, team@3 reaches 4.6%, matching best@13, and team@5 reaches 8.0%, matching best@33. The results support collaboration when feedback and compute are sufficient, not universally.
Key insights
- Communication supports cumulative search: different agents can discover successive improvements and combine them, approximated by summing the fastest discovery time at each stage rather than selecting the fastest complete trajectory.
- The advantage grows with task depth and becomes visible only after a coordination warm-up period; sufficient per-agent compute and reliable intermediate scoring are important prerequisites.
- Communication is not universally beneficial: incomplete verification, premature convergence, shared-state corruption, or overly restricted budgets can make independent sampling perform better.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.