Reproduce “Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruixin Zhang, Wuyang Dai, Hung Viet Pham, Gias Uddin, Jinqiu Yang, Song Wang
An empirical study analyzed 3,864 publicly reported bugs in Claude Code, Codex, and Gemini CLI, categorizing them by type, root cause, symptom, and architectural location. Findings reveal functional bugs are most prevalent (67%), API & Integration Errors are the leading root cause (21.4%), and the Tool/API Orchestration layer accounts for the largest share of defects (37.6%).
Key insights
- A majority of reliability issues in AI coding tools stem from systemic engineering challenges in integration and operationalization, rather than solely from limitations inherent in LLM reasoning.
- Fragility is concentrated in the 'Tool/API Orchestration' and 'Command Execution & Monitoring' architectural layers, emphasizing the complexity of interactions between AI models and external systems.
- Effective AI coding tool development requires a shift in focus towards ensuring the stability and reliability of the surrounding infrastructure, integration points, and operational environment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.