Reproduce “Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Egor Pakhomov, Erik Nijkamp

The authors audit whether MemoryAgentBench’s Conflict Resolution task measures selective forgetting by implementing its newest-statement-wins rule with a frozen ledger that retains the latest fact for each subject–relation pair and answers by lookup or graph traversal. It answers 642/800 questions and 447/600 held-out questions. Yet 67 released gold answers are unreachable under that rule, limiting what aggregate scores establish.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.