Reproduce “CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiwen Yan, Hongcheng Guo, Xuanqing Shi, Shaosheng Cao, Donglin Di, Zhoujun Li
CodeIF introduces the first benchmark specifically designed to assess the instruction-following capabilities of large language models for code generation, revealing that even state-of-the-art models struggle significantly to satisfy complex, multi-faceted coding instructions simultaneously, with a top completely satisfaction rate of 0.362 on hard tasks. It provides a comprehensive evaluation framework across multiple languages and difficulty levels, using novel metrics to measure adherence to diverse coding constraints.
Key insights
- Even leading LLMs like DeepSeek-V3 and Claude-3-5-Sonnet exhibit substantial difficulty in fully satisfying all complex instructions, achieving a maximum Completely Satisfaction Rate (CSR) of only 0.362 on hard tasks.
- Model performance in instruction following generally correlates with increased model scale, though commercial closed-source models consistently outperform open-source counterparts, attributed to superior training data and RLHF techniques.
- Performance significantly degrades with increased task complexity and varies considerably across programming languages, with C++ proving most challenging due to its intricate features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.