Reproduce “StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luanbo Wan, Weizhi Ma
StoryBench introduces a dynamic benchmark utilizing interactive fiction games to evaluate Large Language Models' long-term memory, particularly in knowledge retention and sequential reasoning across multi-turn interactions. It revealed that models like Claude 3.5 Sonnet and GPT-4o exhibit stronger self-correction, but all tested LLMs still struggle with sustained contextual consistency and multi-step error recovery.
Key insights
- Suppressed immediate feedback in "Self Recovery" mode can reveal deeper long-term memory limitations and sometimes improve metrics like "Longest Consecutive Correct Sequence" by forcing models to build more coherent internal narrative understanding.
- Current LLMs, even top performers, primarily exhibit shallow search strategies for error correction, struggling to trace and revise errors across long, multi-step causal chains.
- Overall task success (e.g., completing the story) does not always correlate directly with per-step accuracy, as demonstrated by Claude 3.5 Sonnet completing more stories than Doubao despite slightly lower overall accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.