Reproduce “ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, Jie Tang
Zhipu AI and Tsinghua University developed ComplexFuncBench, a new benchmark for evaluating large language models' complex function calling abilities, focusing on multi-step, constrained, and long-context scenarios. Their evaluation shows leading closed-source models achieve around 60% success rate, while open-source models lag, with "value_error" being a dominant failure mode due to challenges in parameter reasoning from implicit or long contexts.
Key insights
- ComplexFuncBench and ComplexEval establish a more rigorous and nuanced standard for evaluating LLMs' tool use in real-world scenarios, moving beyond simple, single-turn interactions.
- The detailed error analysis, particularly the prevalence of "value_error" and "stop_early" errors, precisely identifies current LLM limitations in parameter value reasoning, information extraction from long contexts, and robust multi-step planning.
- The framework offers actionable insights for future research to improve LLM capabilities in contextual reasoning, self-correction, and handling domain-specific challenges in function calling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.