Reproduce “Lost in Stories: Consistency Bugs in Long Story Generation by LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Li, Xinrui Guo, Yuhao Wu, Roy Ka-Wei Lee, Hongzhi Li, Yutao Xie
Researchers from Microsoft and Singapore University of Technology and Design introduced ConStory-Bench, a benchmark, and CONSTORY-CHECKER, an automated pipeline, to systematically evaluate narrative consistency in long-form stories generated by Large Language Models. Their analysis revealed that consistency errors accumulate linearly with output length, frequently occurring in factual and plot logic categories, and correlate with higher token-level uncertainty.
Key insights
- Consistency errors tend to accumulate approximately linearly with the output length of generated narratives across different LLM architectures, though the rate varies by model.
- Text segments containing errors consistently exhibit significantly higher token-level Shannon entropy (lower confidence, higher unpredictability) compared to consistent text, suggesting a link between internal model uncertainty and error manifestation.
- Factual & Detail Consistency errors frequently co-occur with Characterization, World-building & Setting, and Timeline & Plot Logic errors, while Narrative & Style errors appear to arise from distinct mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.