Reproduce “ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanhang Zheng, Peng Li, Wei Liu, Yang Liu, Jian Luan, Bin Wang
ToolRerank is a tool retrieval framework designed to enhance Large Language Models' ability to use external tools by improving the selection of relevant APIs. It introduces adaptive truncation to differentiate between seen and unseen tools and employs hierarchy-aware reranking strategies to leverage the hierarchical structure of tool libraries, leading to improved retrieval accuracy on the ToolBench dataset, particularly for unfamiliar tools.
Key insights
- Retrieval quality for known tools is consistently higher and benefits from stricter candidate filtering, while performance for unfamiliar tools improves when more candidates are considered by the reranker.
- Leveraging the hierarchical structure of tool libraries (tools containing multiple APIs) and distinguishing between single-tool and multi-tool queries allows for more effective reranking strategies.
- Differentiating retrieval and reranking strategies based on whether tools are 'seen' (in training data) or 'unseen' (not in training data) significantly improves overall performance, especially for unseen tools.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.