Reproduce “AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Du, Fangyun Wei, Hongyang Zhang
AnyTool enables large language models to effectively utilize over 16,000 real-world APIs by employing a hierarchical API retriever for efficient search and a self-reflection mechanism for iterative improvement. The system achieved an average pass rate improvement of +35.4% over ToolLLM on the ToolBench benchmark, demonstrating robust performance on a vast API collection.
Key insights
- Organizing APIs into a hierarchical structure significantly improves an LLM's efficiency in searching and utilizing large API collections by managing context length.
- A self-reflection mechanism, allowing the LLM to analyze failures and learn, consistently increases performance, with most improvements occurring within the first few reflection iterations.
- A revised evaluation protocol that considers only solvable queries provides a more accurate assessment of an agent's real-world API utilization capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.