Reproduce “You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong, Siying Hu, Haichang Li, Ziying Wang, Zhicong Lu

This research introduces a diagnostic benchmark and a 4,735-item human-validated dataset to evaluate large language models' ability to interpret fine-grained social pragmatic meanings in indirect and playful Chinese online comments. It quantifies current LLM limitations, demonstrating an average leave-writer-out accuracy of 68.70% compared to human accuracy of 90.8%, and confirms the critical role of multi-layered context in such interpretations.

Key insights

  • LLMs exhibit a substantial gap in performing robust, fine-grained social pragmatic inference, particularly struggling with specific targets, stances, and underlying mechanisms of playfulness, even when identifying general humor.
  • The presented diagnostic methodology, which incorporates LLM-generated questions and human validation for tasks without objective external anchors, offers a systematic approach to expose model misunderstandings in complex social contexts.
  • Context is critically important for understanding indirect and playful language, with ablations showing a 39.07 percentage point mean accuracy drop when context is removed, and different contextual layers (post-level, local interaction, direct parent) each contributing to comprehension.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.