Reproduce “GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov

GitGoodBench introduces a new benchmark for evaluating AI agents' performance on core Git operations, including Merge Conflict Resolution, Interactive Rebase, and Iterative Committing. The benchmark comprises diverse datasets mined from open-source repositories and shows that a GPT-4o baseline achieves a 21.11% solve rate across these complex version control tasks.

Key insights

  • Proficiency in programming does not automatically translate to proficiency in Git VCS operations; current advanced LLMs (like GPT-4o) exhibit significant limitations on these tasks, even with specialized tooling.
  • The complexity of Git tasks for AI agents correlates with the level of abstraction from direct code generation, with MCR (requiring code patches) proving more challenging than IR or ICC (modifying structured plans).
  • The proposed difficulty heuristic for Merge Conflict Resolution (MCR) accurately reflects empirical agent performance, with solve rates dropping from 22.58% for easy conflicts to 0% for hard ones.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.