Reproduce “InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

InterCode establishes a standardized benchmark and framework for evaluating Large Language Models (LLMs) in interactive coding scenarios with execution feedback, formalizing the process as a Reinforcement Learning environment. Experiments demonstrate that interaction significantly boosts LLM performance, with GPT-4's success rate on InterCode-SQL increasing from 9.1% (static) to 73.7% (interactive), highlighting the value of iterative refinement while also exposing current limitations in long-term context utilization.

Key insights

  • LLMs benefit substantially from interactive execution feedback, allowing them to discover context, correct errors, and perform modular problem-solving, significantly improving success rates compared to static generation.
  • Current LLMs still struggle with long-term memory, effective integration of cumulative observations, and recognizing unproductive lines of reasoning in extended interactive sessions, leading to performance plateaus.
  • Explicit prompting strategies like ReAct and Plan & Solve can enhance LLMs' reasoning and problem-solving, though their effectiveness varies by task, indicating a need for more adaptive reasoning mechanisms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.