Reproduce “FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Sunghee Jung, Myeongcheol Shin
Kakao Corp. researchers developed FunctionChat-Bench, a benchmark for comprehensively evaluating language models' generative capabilities in Korean tool-use dialogs, covering both technical tool calls and conversational responses. This evaluation revealed that models performing well in single-turn tool calls may struggle with multi-turn conversational aspects like slot questions and relevance detection, while advanced models show robust performance across all interaction types.
Key insights
- Models' performance in single-turn tool calls does not directly translate to their overall generative capabilities in complex, multi-turn tool-use dialogs.
- Generating appropriate conversational outputs, such as asking clarifying Slot Questions or correctly identifying Relevance, is critical for effective user interaction and overall model performance.
- The number of candidate functions and their semantic similarity influence tool call accuracy, with some models exhibiting unexpected improvements as more functions are introduced.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.