Reproduce “Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hiroshi Nonaka, K. E. Perry
A quantitative methodology is introduced to evaluate LLM story generation by analyzing the social structures within narratives using signed character networks. This analysis reveals that LLMs consistently produce stories with high-density, overwhelmingly positive social interactions, contrasting with the more complex and conflict-rich relationships found in human-written works.
Key insights
- LLMs exhibit a consistent and pronounced bias towards creating narratives centered on overwhelmingly positive character relationships and tightly-knit social groups.
- Human-written stories feature significantly more complex and varied social dynamics, including more intricate and frequent negative interactions, which contribute to dramatic depth and conflict.
- Network analysis metrics can effectively quantify fundamental differences in storytelling patterns and structural biases between AI-generated and human-authored narratives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.