Reproduce “AutoDev: Automated AI-Driven Development” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, Neel Sundaresan
AutoDev is an automated AI-driven development framework from Microsoft that empowers AI agents to autonomously plan and execute complex software engineering tasks within a secure, interactive environment. It achieved a 91.5% Pass@1 score on HumanEval for code generation and 87.8% for test generation, significantly improving upon zero-shot GPT-4 performance.
Key insights
- Providing Large Language Models (LLMs) with an interactive environment and a comprehensive suite of tools substantially improves their performance in complex software engineering tasks beyond simple inference.
- Autonomous AI agents can effectively plan, execute, and self-correct through iterative interaction with a codebase and its feedback mechanisms (e.g., test results, compiler errors).
- A secure and controlled development environment, such as Docker containers with explicit permissions, is crucial for deploying autonomous AI agents in practical software development workflows.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.