Reproduce “ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunkun Wang, Yue Zhang, Zhen Qin, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, Shuiguang Deng

ExploraCoder, a training-free framework, enables Large Language Models to generate code utilizing multiple unseen APIs by integrating task planning with a chained exploratory execution process that learns from intermediate code outputs. It demonstrated substantial improvements, achieving an an absolute Pass@1 increase of 14.46% for API-pretrained GPT-4-1106-preview on the Torchdata-Manual benchmark and a 19.8% absolute Pass@20 gain over naive RAG for API-untrained models.

Key insights

  • Decomposing complex multi-API problems into manageable subtasks and iteratively validating API usage through execution feedback (CoAE) is crucial for LLMs to overcome hallucination and documentation ambiguities.
  • Intermediate self-debugging at the subtask level significantly improves the robustness and correctness of the final generated code by rectifying errors early in the process.
  • Guiding the LLM's task planning with library-level in-context learning, including high-level summaries and few-shot examples, effectively aligns the LLM's understanding with the target API's functional granularity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.