Reproduce “DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kyochul Jang, Donghyeon Lee, Kyusik Kim, Dongseok Heo, Taewhoo Lee, Woojeong Kim, Bongwon Suh
DICE-BENCH evaluates function calling when the task, parameters, and dependencies are distributed across several speakers and dialogue rounds rather than stated in one prompt. Across 19 language models, exact-match performance generally declines as conversations become longer and more information-dispersed, with GPT-4o falling from 74.1% in the first round to 59.2% in the fourth; the benchmark also introduces DICE-SCORE to quantify this dispersion.
Key insights
- Longer and more fragmented conversations reliably make tool selection and parameter extraction harder: GPT-4o drops from 74.1% exact match in Round 1 to 59.2% in Round 4, with similar trends for Gemini 2 Flash.
- General-purpose conversational models often outperform tool-specialized models, suggesting that training on isolated API calls does not transfer well to multi-party context tracking.
- Eristic conversations are particularly difficult because interruptions, shifting positions, and emotionally charged exchanges obscure the stable task information needed for a call.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.