Reproduce “ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyi Xu, Yiwen Hu, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Ji-Rong Wen
The paper introduces ICPC-Eval, a new benchmark of 118 problems from recent International Collegiate Programming Contests to evaluate advanced Large Language Models (LLMs) in complex algorithmic reasoning. It includes a novel LLM-driven test case generation pipeline and the Refine@K metric, demonstrating that top LLMs achieve a Refine@5 score of up to 28.8% and struggle with domains like computational geometry and search algorithms.
Key insights
- ICPC-Eval effectively sets a substantially higher bar for LLM reasoning, revealing a wide performance gap between current LLMs and human competitive programmers.
- The Refine@K metric accurately captures the reflective capabilities of LLMs, as reasoning-focused models consistently improve their solutions with execution feedback, unlike non-reasoning models.
- LLM performance varies significantly across algorithmic domains, with computational geometry and search algorithms posing the most formidable challenges, indicating specific areas for improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.