Reproduce “KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongjun Kim, Chanhee Park, Chanjun Park, Heuiseok Lim
KITE introduces the first dedicated benchmark for evaluating large language models' open-ended instruction-following abilities in Korean, developed by researchers from Korea University and Soongsil University. It reveals that GPT-4o achieves 64.46% accuracy on Korean-specific tasks, significantly outperforming existing Korean-specific LLMs, and demonstrates a high correlation with human judgment (r=0.93-0.96).
Key insights
- Instruction-following is a distinct LLM capability, separate from general language understanding or factual knowledge, requiring specialized tuning.
- A substantial performance gap exists between highly capable general-purpose LLMs (like GPT-4o) and Korean-specific models on nuanced Korean instruction-following tasks.
- The impact of few-shot examples on instruction-following performance can be inconsistent for many models, though advanced models like GPT-4o exhibit robustness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.