Reproduce “Academically intelligent LLMs are not necessarily socially intelligent” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruoxi Xu, Hongyu Lin, Xianpei Han, Le Sun, Yingfei Sun
Large Language Models often lack social intelligence despite excelling in academic tasks, prompting the development of the Situational Evaluation of Social Intelligence (SESI) benchmark. This new evaluation, derived from real-world social scenarios, reveals that current models perform at 55.2% accuracy and that social intelligence is a distinct capability from their academic abilities, often hindered by a 'superficially friendly' response pattern.
Key insights
- LLM social intelligence is a distinct capability from academic intelligence, showing low correlation with performance on knowledge, reasoning, and comprehension benchmarks.
- Current LLMs often exhibit 'superficial friendliness,' providing generic, positive responses that fail to adapt to the specific social context or severity of the situation.
- LLMs do not appear to possess a conceptual understanding of social intelligence, as explicit prompts to exhibit higher social intelligence paradoxically led to decreased performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.