Reproduce “SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyu Fan, Kirill Vasilevski, Dayi Lin, Boyuan Chen, Yihao Chen, Zhiqing Zhong, Jie M. Zhang, Pinjia He and 1 more
The SWE-Effi framework introduces a comprehensive evaluation for AI software engineering systems, integrating resource consumption such as LLM tokens, CPU time, and monetary cost with traditional resolution rates. This work reveals critical inefficiencies in current agent designs, demonstrating that optimal performance requires balancing task success with resource effectiveness for practical deployment.
Key insights
- The overall effectiveness of a SWE AI system is highly dependent on the underlying LLM, often more so than the scaffold design, with resolution rates alone being insufficient to gauge true performance.
- Higher-quality reasoning from specialized LLMs can lead to paradoxically lower overall token consumption by minimizing the total number of iterations and API calls required to solve an issue.
- A 'Token Snowball Effect' occurs in many agent frameworks due to naive memory accumulation, where input token consumption grows disproportionately with API calls, hindering reasoning and increasing costs.
- Unresolved issues are consistently far more resource-intensive than successful ones, highlighting a lack of 'futility detection' in current AI systems, leading to expensive and unproductive trajectories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.