Reproduce “SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang and 2 more
A new benchmark for Chinese Large Language Models (LLMs) called SuperCLUE provides a comprehensive, user-preference-centric evaluation framework for the Chinese language. It reveals a substantial performance gap between top global models and Chinese LLMs, validates GPT-4 as a reliable automatic evaluator for open-ended Chinese responses, and demonstrates that combining open-ended and closed-ended evaluations significantly improves the prediction of actual user preferences.
Key insights
- GPT-4 consistently demonstrated superior performance over all evaluated Chinese LLMs across all SuperCLUE sub-tasks.
- Closed-ended questions proved insufficient to reflect real user preferences in open-ended scenarios, showing no significant correlation with user feedback and low discriminative power among models.
- An advanced LLM (GPT-4) can reliably serve as an automatic judge for open-ended Chinese responses, exhibiting 80% Pearson correlation with human evaluations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.