Reproduce “ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heming Xia, Yongqi Li, Cunxiao Du, Mingbo Song, Wenjie Li
TOOLSPEC introduces a training-free speculative decoding method that combines schema-aware drafting and retrieval-augmented speculation to accelerate tool-calling generation in Large Language Models. It achieves speedups of 3.5x to 4.2x compared to standard autoregressive decoding and up to a 71% relative improvement over prior plug-and-play techniques across various LLMs and benchmarks.
Key insights
- Tool-calling generation, not execution, is a dominant latency bottleneck in many LLM-agent scenarios, accounting for up to 96% of total inference time for large models.
- Tool-calling outputs possess strong structural regularities due to adherence to JSON schemas and frequently exhibit recurring invocation patterns, making them highly amenable to specialized drafting.
- A hybrid drafting mechanism that combines deterministic schema-guided token generation with the reusability of historical call patterns significantly improves draft quality and acceptance rates in speculative decoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.