Reproduce “LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mateusz Winiarek, Maksymilian Bilski, Mateusz Jacniacki
The LoSoNA benchmark assesses the capability of Large Language Models (LLMs) to infer and adapt to implicit local conversational norms in multi-party group chats. It reveals that while most models struggle with this task under naive prompting, certain frontier models like Gemini 3.1 Pro and Claude Fable 5 achieve significant performance gains, reaching over 80% accuracy-at-3, when explicitly prompted for norm awareness.
Key insights
- Most LLMs, when provided with naive prompting, generally fail to infer and adapt to implicit local social norms in group conversations, indicating a gap in current social intelligence capabilities.
- Explicitly informing models about the potential relevance of local patterns or norms can significantly improve performance for certain frontier models, but is not a universally effective intervention, sometimes even causing regression.
- The benchmark offers a first-person, behavioral assessment of social adaptation, contributing to a more dynamic and functionally-oriented evaluation of Theory of Mind in LLMs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.