Reproduce “Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chuanrui Hu, Tong Li, Xingze Gao, Hongda Chen, Yi Bai, Dannong Xu, Tianwei Lin, Xiaohong Li and 3 more
EverMemBench evaluates memory in simulated workplace collaboration across five projects. It tests fact recall, applying remembered rules, and modeling people’s styles, skills, and roles across group chats. Gemini-3-Flash answers 26.51% of multi-hop questions with full context and 88.37% with gold evidence, while LLaMA-4-Scout reaches 37.35% with oracle evidence, showing that evidence access and reasoning remain challenges in this benchmark.
Key insights
- Simply scaling context length or improving similarity-based retrieval is insufficient; multi-party attribution, temporal revision, and inferential relevance introduce structural challenges that current memory paradigms struggle to address.
- LLMs and memory systems exhibit a 'multi-hop reasoning collapse' when evidence is fragmented across speakers and groups, and struggle significantly with temporal reasoning involving event lifecycles and revisions.
- Retrieval-centric memory systems are limited in capturing emergent, implicit persona traits like communication style, which are never explicitly stated and require aggregating distributed behavioral signals over long interaction histories.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.