Reproduce “CHATTER: A Character Attribution Dataset for Narrative Understanding” on OpenResearch

About this paper

Sabyasachee Baruah, Shrikanth Narayanan

Researchers at the University of Southern California developed CHATTER, the largest character attribution dataset utilizing narrative tropes, alongside CHATTEREVAL, a human-validated benchmark. Evaluations with large language models demonstrated their substantial reliance on prior knowledge for character attribution, with top closed-source models reaching up to 81.9% accuracy when leveraging this pre-trained information.

Key insights

• Narrative "tropes" provide a scalable, well-defined, and discrete framework for character attribution, necessitating document-level understanding from models.
• Large language models (LLMs) heavily leverage their pre-trained knowledge of characters for attribution, often outperforming in-context reasoning from provided anonymized texts.
• Character anonymization during evaluation is critical to prevent LLMs from relying on prior knowledge and to assess their true in-context narrative comprehension.
• Focusing on character-relevant narrative segments is often as effective as processing full documents, indicating potential for efficient long-context processing.

Summary by alphaXiv.