Reproduce “MemConflict: Evaluating Long-Term Memory Systems Under Memory Conflicts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhen Tao, Jinxiang Zhao, Peng Liu, Dinghao Xi, Yanfang Chen, Wei Xu, Zhiyu Li
A diagnostic framework called MemConflict assesses the reliability of long-term memory systems for conversational agents by systematically constructing dynamic, static, and conditional memory conflicts within multi-session user histories. Evaluations revealed current memory systems exhibit uneven strengths across conflict types, with static conflicts being particularly difficult, and performance declining as interaction history length and conflict distance increase.
Key insights
- Memory systems require diagnostic evaluation beyond outcome-level performance to understand how they retrieve and rank evidence under diverse conflict types.
- A "fitness-for-use" perspective, defining memory validity based on temporal, factual, and contextual applicability, is essential for robust long-term memory.
- White-box analysis of memory retrieval and ranking reveals that gold memories may be retrieved but not top-ranked, leading to utilization failures in final answer generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.