Reproduce “PSB2: The Second Program Synthesis Benchmark Suite” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thomas Helmuth, Peter Kelly

Hamilton College researchers introduce PSB2, a new benchmark suite of 25 program synthesis problems curated from sources like coding challenges and academic assignments. Designed to be more challenging and standardized than its predecessor, PSB1, initial PushGP experiments demonstrate increased difficulty by solving 13 problems and confirming high generalization rates for successful solutions.

Key insights

  • Standardized and continually updated benchmarks are critical for driving progress and facilitating fair comparisons in program synthesis.
  • Problems selected for benchmarks must be difficult enough to provide "room for improvement" for future algorithms, but not intractably hard for current methods.
  • Robust generalization to unseen data, not just training set performance, is essential for evaluating program synthesis systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.