Reproduce “Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yeeun Kim, Young Rok Choi, Eunkyung Choi, Jinhwan Choi, Hai Jin Park, Wonseok Hwang
KBL, the first pragmatic and expert-verified benchmark for Korean legal language understanding, evaluates large language models across diverse legal knowledge, reasoning, and bar exam tasks in both closed-book and retrieval-augmented generation settings. Evaluations on KBL indicated that current LLMs, including GPT-4, exhibit substantial limitations in complex Korean legal scenarios, despite some open-source models outperforming GPT-3.5, and RAG's effectiveness varying significantly.
Key insights
- Current LLMs, including advanced commercial models like GPT-4, exhibit significant limitations in complex Korean legal understanding, particularly on the Korean Bar Exam, with GPT-4 achieving only 48.1%.
- Open-source LLMs like Qwen2-72b can surpass GPT-3.5 on certain Korean legal knowledge and reasoning tasks, demonstrating progress in localized model capabilities.
- The effectiveness of Retrieval-Augmented Generation (RAG) is inconsistent, varying based on the specific LLM, task type, and corpus, sometimes improving performance but occasionally causing degradation, especially in specific bar exam domains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.