Reproduce “SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hexuan Deng, Yue Wang, Wenyu Jiang, Cheng Yang, Haolin Yang, Zhaohua Zhang, Chenchen Zhao, Beiduo Chen and 8 more
SWE-Journey evaluates coding assistants on long sequences of repository changes while interacting with simulated users whose requirements and behaviors differ. It combines progressively built tasks with multi-turn persona-conditioned conversations, measuring requested functionality and preserved behavior. Across 13 models, mean functionality-test pass rates were 78.50% with the Software Architect persona and 23.00% with the Non-coder persona; these conditions do not isolate expertise.
Key insights
- Interaction conditions strongly shape measured performance: expert, repository-level guidance is associated with much higher scores than symptom-based, non-technical requests.
- Interaction is not uniformly beneficial or harmful. In an 11-model comparison, single-shot instructions yielded lower mean final F2P than the Software Architect persona but higher results than the other three personas.
- Models preserved existing behavior more successfully than they implemented all requested changes, and final F2P exceeded process-time F2P, indicating that some issues were fixed later in a session.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.