Reproduce “UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peijun Qing, Fobo Shi, Soroush Vosoughi
UTILMEM evaluates whether long-term conversational memory systems can integrate distributed evidence into grounded, useful responses rather than merely retrieve isolated facts. Across 1,717 long-context tasks in five domains, the benchmark shows that compression-based memories and semantically similar distractors can substantially reduce answer quality even when relevant sessions are retrieved.
Key insights
- Memory utilization is distinct from memory retrieval: even with at least 80% of the relevant sessions retrieved, systems remain below oracle quality, with high-recall performance varying from 6.4 RS in mental wellness to 7.9 in finance.
- Aggressive fact-oriented compression can discard contextual information needed for later synthesis; graph augmentation provides little recovery for the weakest compression-based systems.
- A small number of semantically similar distractors causes most of the performance loss, and larger language models improve absolute scores without removing this sensitivity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.