Reproduce “GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Chaudhary and 4 more
GitChameleon 2.0 asks whether generated Python code works with a project’s pinned library version. The benchmark contains 328 problems across 26 libraries, each evaluated with visible and hidden tests. It measures hidden-test success. Under greedy decoding, o1 achieves 51.2% success; retrieval-augmented generation raises GPT-4.1’s rate from 48.5% to 58.5%. These results show version-conditioned generation remains challenging in this Python benchmark.
Key insights
- Current state-of-the-art AI code generation systems achieve modest success rates (around 50%) on version-conditioned problems, indicating a critical limitation in handling library incompatibilities.
- Integrating execution feedback through self-debugging or a code execution sandbox is crucial for substantially improving functional accuracy across various models and agentic systems.
- While Retrieval-Augmented Generation (RAG) can provide performance boosts, access to documentation alone is insufficient to solve a significant portion of version incompatibility challenges, and models found new feature additions particularly challenging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.