Reproduce “Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Justin Waugh
Approximate Labs introduced "Pencil Puzzle Bench," a benchmark with 62,231 pencil puzzles providing step-level verification to evaluate multi-step reasoning in large language models. The benchmark demonstrated that iterative verification significantly improves model performance, with the best model achieving 56.0% success in agentic mode, and revealed solution compressibility as a key difficulty predictor.
Key insights
- There exists a significant "agentic gap," where iterative verification and course-correction through step-level feedback substantially boost LLM performance, particularly for models with weaker direct-ask capabilities.
- Reasoning depth (internal deliberation) and agentic iteration (external verification and course-correction) represent distinct, complementary axes of LLM capability.
- Solution compression ratio is a significantly better predictor of puzzle difficulty (explaining 4.2x more variance) than the mere number of moves.
- Systematic server-side timeouts observed at "xhigh" reasoning effort for frontier models indicate practical limits on the duration of single inference requests, highlighting infrastructure constraints for highly deliberative LLM systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.