Reproduce “HeartBench: Probing Core Dimensions of Anthropomorphic Intelligence in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiaxin Liu, Peiyi Tu, Wenyu Chen, Yihong Zhuang, Xinxia Ling, Anji Zhou, Chenxi Wang, Zhuo Han and 4 more
HeartBench introduces a comprehensive benchmark designed to evaluate the anthropomorphic intelligence of Chinese Large Language Models, uncovering that even top-performing models achieve only about 60% of expert-defined ideal scores. It utilizes authentic psychological counseling scenarios and a validated rubric-based evaluation system to assess emotional, social, and ethical capabilities.
Key insights
- Current state-of-the-art LLMs achieve an average of only approximately 60% of the expert-defined ideal score, indicating a substantial gap in anthropomorphic intelligence.
- Models demonstrate universal deficits in dimensions requiring advanced cognitive simulation like curiosity, humor, and proactiveness, while exhibiting model-specific strengths in other areas.
- A "Hard Set" of challenging scenarios revealed a significant average performance decay of 37.8% and caused a reshuffling of top-performing models, highlighting limitations in complex, nuanced interactions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.