Reproduce “SimpleTool: Parallel Decoding for Real-Time LLM Function Calling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoxin Shi, Jiaxin Wan, Linkang Dong, Wei Jiang, Yue Liu, Zengfeng Huang
SimpleTool introduces a method for real-time Large Language Model (LLM) function calling by combining special tokens for output compression and parallel decoding. This approach achieved a 3-6x end-to-end speedup, enabling sub-100 ms latency for models up to 7B parameters on consumer GPUs, and surpassed FunctionGemma in both accuracy and tail latency for edge function calling.
Key insights
- LLM-generated function calls exhibit weak causal dependencies among arguments and substantial token redundancy, making them amenable to aggressive compression and parallelization.
- Jointly exploiting token compression via special tokens and parallelizing independent decoding streams can achieve significant end-to-end acceleration for real-time function calling.
- Running multiple parallel decoding streams incurs negligible computational overhead on modern GPUs by utilizing otherwise-idle compute capacity during the memory-bandwidth-bound decode phase.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.