Reproduce “Episodic Memories Generation and Evaluation Benchmark for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexis Huet, Zied Ben Houidi, Dario Rossi
Researchers from Huawei Technologies Co., Ltd. introduce a new benchmark for evaluating episodic memory in Large Language Models, leveraging synthetic data to ensure contamination-free assessment. The study demonstrates that current LLMs struggle with multi-event recall, chronological ordering, and preventing confabulation, even with advanced models and RAG strategies.
Key insights
- Current LLMs exhibit a 'cue overload' phenomenon, where performance declines as more events match a query, indicating difficulty in distinguishing specific information from similar experiences.
- Naive fine-tuning on single-event data only leads to memorization and does not enable generalization to deeper episodic understanding or multi-event reasoning.
- LLMs struggle significantly with chronological ordering and maintaining entity states over time, indicating a fundamental challenge in processing and reasoning about temporal relationships between events.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.