Reproduce “From Software History to Executable Regression Tasks: A System for Evaluating Software Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye
EReg-Factory turns historical introducing–fixing commit pairs and tests into replayable regression instances, reconstructing both environments and reviewing paired outcomes against code changes. It provides validated tests, execution evidence, root locations, and repairs for three evaluations. On 83 manually selected instances, repair and test-generation success reached 68.7% and 77.1%; exact root-line matching remained at or below 20.5%. Evaluation covers Python/pytest instances only.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.