Reproduce “TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryo Fujii, Makoto Morishita, Kazuki Yano, Jun Suzuki
Researchers introduced TimeMachine-bench, the first scalable and continuous benchmark for evaluating Large Language Models on repository-level software migration tasks in Python, revealing high success rates for state-of-the-art models but also significant challenges in producing necessary edits and handling task complexity. The benchmark's date-based environment control enables automated generation of diverse migration scenarios for real-world projects.
Key insights
- State-of-the-art LLMs, like Claude Sonnet 4, achieved high success rates (99.0% pass@1) on human-verified migration tasks, demonstrating strong capabilities in adapting code to dependency changes when given sufficient interaction budget.
- Despite high success rates, LLMs struggle to produce only necessary code edits, with top models like Claude Sonnet 4 showing a prec@1 of only 78.0%, indicating a tendency for redundant modifications.
- Performance consistently declines with increased task difficulty, revealing that complex, hard-category migration scenarios remain a significant challenge for all evaluated LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.