Reproduce “RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, Roshanak Zilouchian Moghaddam
Researchers from UC Berkeley and Microsoft introduce RefactorBench, a new benchmark designed to evaluate language model agents on multi-file code refactoring tasks requiring stateful reasoning and long-horizon planning. The study reveals specific failure modes in current agents and demonstrates that state-aware interfaces can increase task resolution by 43.9% relative to baselines.
Key insights
- Current LM agents exhibit novel failure modes in multi-file refactoring, including difficulty in finding relevant edit locations, inability to tolerate erroneous intermediate code states, and context flooding that dilutes long-term objectives.
- The performance gap between LLM agents (12-35% success) and human developers (87% success) on complex refactoring tasks is substantial, highlighting limitations in compositional reasoning and state management.
- Explicitly managing and providing an accumulated state to the agent significantly mitigates "forgetting" issues over long trajectories, improving decision-making and long-term planning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.