Reproduce “INSEva: A Comprehensive Chinese Benchmark for Large Language Models in Insurance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shisong Chen, Qian Zhu, Wenyan Yang, Chengyi Yang, Zhong Wang, Ping Wang, Xuan Lin, Bo Xu and 10 more
INSEva is a comprehensive Chinese benchmark created to evaluate Large Language Models in the insurance domain. It features 38,704 examples across diverse insurance business areas, task formats, and cognitive dimensions, demonstrating that current LLMs grasp foundational concepts but struggle with complex reasoning and compliance, exhibiting a trade-off between faithfulness and completeness in open-ended responses.
Key insights
- Current LLMs possess a foundational understanding of insurance but demonstrate significant gaps in complex logical reasoning (ILR) and regulatory compliance (ISC) tasks.
- A crucial trade-off exists in open-ended responses between generating faithful (non-hallucinatory) content and achieving comprehensive coverage of ground truth, with faithfulness being critical in high-risk insurance applications.
- General-purpose LLMs frequently outperform existing domain-specific financial LLMs in insurance-related tasks, highlighting the distinct needs of the insurance domain compared to broader finance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.