Reproduce “SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fake Lin, Binbin Hu, Xi Zhu, Ziwei Zhao, Zhi Zheng, Ziqi Liu, Zhiqiang Zhang, Jun Zhou and 1 more

SmellBench, a new benchmark designed to evaluate code agents on repository-level refactoring, proactively injects code smells into clean code. Experiments with state-of-the-art models show low smell elimination scores (best 50.34%) despite high functional correctness, underscoring agents' limitations in structural reasoning and multi-file coordination.

Key insights

  • Evaluating refactoring requires multi-dimensional metrics beyond functional test pass rates, as agents can maintain correctness without improving code quality or structure.
  • Current code agents demonstrate significant limitations in autonomous code smell localization and handling complex, multi-file coordinated refactoring tasks.
  • The performance of models decreases with increasing task difficulty, and while stronger LLMs show some resilience, even the best combinations still struggle to achieve high smell elimination rates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.