Reproduce “Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents” on OpenResearch

About this paper

Yifei Li, Weidong Guo, Lingling Zhang, Rongman Xu, Muye Huang, Hui Liu, Lijiao Xu, Yu Xu and 1 more

LoCoMo-Plus tests whether conversational systems carry implicit personal constraints from earlier dialogue into later responses, rather than merely recalling explicit facts. The authors construct long-dialogue cases with cue–trigger semantic disconnect and evaluate whether responses acknowledge or adapt to cues. Gemini-2.5-Pro scores 26.06 on LoCoMo-Plus versus 71.78 on LoCoMo’s five-category factual average, a reported 45.72-point gap; these are different tasks.

Key insights

• Cognitive memory, which involves retaining and applying implicit constraints, presents a persistent and significant challenge for all evaluated LLMs, including state-of-the-art and memory-augmented systems.
• Empirical evidence confirms that explicit task disclosure in prompts and length-biased generation metrics can substantially distort measured LLM performance and undermine evaluation reliability.
• The LLM-as-a-judge approach for constraint-consistency evaluation demonstrates high reliability, showing strong alignment with human judgment and stability across different judge models.

Summary by alphaXiv.