Reproduce “Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xin-Ye Li, Jiang-Tian Xue, Zheng Xie, Ming Li

The BRAINSTORM framework enhances Large Language Models' (LLMs) ability to solve complex, competition-level programming problems by introducing a human-inspired brainstorming and thought selection phase before code generation. It boosted ChatGPT's pass@100 on CodeContests by 61.0% to 29.3% and enabled an overall pass@1 of 40.3% on LeetCode, matching human programmer performance.

Key insights

  • Explicitly structuring an LLM's problem-solving process with a brainstorming and thought selection phase significantly improves its algorithmic reasoning for complex code generation.
  • A learned ranker can effectively evaluate and select high-quality algorithmic blueprints from multiple LLM-generated ideas, enhancing the quality of subsequent code generation.
  • While beneficial for complex tasks, the intermediate reasoning steps introduced by brainstorming can be detrimental to performance on simpler, more straightforward programming problems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.