Reproduce “NarraBench: A Comprehensive Framework for Narrative Benchmarking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sil Hamilton, Matthew Wilkens, Andrew Piper
A comprehensive framework, NARRABENCH, offers a theory-informed taxonomy for evaluating Large Language Models' narrative understanding across four dimensions: Story, Narration, Discourse, and Situatedness. This work surveys existing benchmarks, revealing that only 27% of defined narrative tasks are well-captured, and proposes new evaluation criteria to foster more nuanced assessment.
Key insights
- Human narrative understanding is fundamentally multi-dimensional, extending beyond 'what happened' to include 'how it was told,' 'who speaks,' and 'why it was told' (Discourse, Narration, Situatedness), which are often overlooked in current LLM evaluation.
- Narrative interpretation is inherently subjective and often requires perspectival or consensus-based judgments rather than a single deterministic answer, challenging the prevailing 'ground truth' methodologies in AI benchmarking.
- A unified, theory-informed taxonomy is essential for systematically categorizing and defining narrative understanding abilities, providing a much-needed framework to guide the development of more comprehensive and nuanced LLM evaluations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.