Reproduce “TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pengfei He, Zhenwei Dai, Bing He, Hui Liu, Xianfeng Tang, Hanqing Lu, Juanhui Li, Jiayuan Ding and 5 more
A new benchmark called TRAJECT-Bench introduces a rigorous framework for evaluating Large Language Model (LLM) agents' tool-use capabilities using over 1,000 real-world APIs, complex multi-step trajectories, and both direct and indirect user queries. Evaluations reveal LLMs struggle significantly with inferring intent from indirect queries and scaling tool use, though agentic frameworks like ReAct provide modest improvements.
Key insights
- LLMs face substantial challenges in interpreting complex user intents from indirect queries and in scaling their tool-use capabilities for longer, more numerous tool sequences.
- Current retrieval-augmented generation methods for tool selection often prove insufficient for complex, implicit queries, struggling to retrieve necessary tools.
- Agentic frameworks like ReAct and dynamic tool retrieval modestly improve tool-use performance, yet fundamental issues in precise tool selection and parameterization persist across state-of-the-art models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.