Reproduce “ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use” on OpenResearch

About this paper

Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu and 6 more

ToolHop introduces a query-driven benchmark for evaluating Large Language Models in multi-hop tool use, generating 995 complex queries and 3,912 executable tools. The evaluation of 14 LLMs reveals that even top models like GPT-4o achieve only 49.04% accuracy, highlighting significant challenges in multi-step reasoning and robust error handling.

Key insights

• Multi-hop tool use remains a substantial unsolved problem for current LLMs, with even state-of-the-art models struggling significantly, demonstrating a critical need for further research.
• Different LLM architectures exhibit distinct strengths and weaknesses in tool use; some struggle with robust tool invocation and CoT reasoning, while others prioritize efficiency (parallel calls) at the cost of accuracy in interdependent steps.
• The ability of LLMs to self-correct during multi-hop tasks is highly dependent on the richness and detail of the diagnostic feedback provided by the tools, emphasizing the importance of robust error handling in tool design.

Summary by alphaXiv.