Reproduce “WizardCoder: Empowering Code Large Language Models with Evol-Instruct” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma and 2 more

WizardCoder, a Code Large Language Model, leverages a tailored instruction-tuning method called Code Evol-Instruct to generate complex programming tasks. This approach enables open-source models to achieve state-of-the-art performance, surpassing GPT-3.5 on HumanEval+ and leading all open-source baselines across multiple code generation benchmarks.

Key insights

  • Instruction complexity, rather than just data quantity, is the primary factor driving performance gains in Code LLMs when trained with evolved data.
  • There exists an optimal number of evolution rounds, indicating a peak level of instruction complexity beyond which further evolution may not yield performance benefits.
  • The improved performance is attributed to the models learning more generalized problem-solving abilities from complex instructions, not from increased similarity or overfitting to test data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.