Reproduce “MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guorui Chen, Tao Liu, Zhenzhen Huang, Yidan Zhang, Liyu Jiang, Shuoyuan Wang, Hanchen Sun, Xu He and 4 more
The authors introduce MemDev-Bench, which tests whether repository history helps or misleads coding agents. It pairs later tasks with earlier, leakage-screened repository tasks labeled for relevance, then evaluates memory across history conditions. On 133 Improvement Tasks, Claude-Mem with Repo History resolves 42.6%, versus 35.3% for NoMem; memory can also harm otherwise solvable tasks in this tested setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.