Reproduce “Training Small Language Models for Transferable Tool Selection” on OpenResearch

About this paper

Zhaoyi Zhou, Akshayaa Magesh, Jeongyeol Kwon, Artem Kuriksha, Daniel Jiang, Andrea Zanette, Ayush Jain

The Plan–Retrieve–Select workflow handles multi-part requests by splitting them into independent substeps, retrieving BM25 candidates, and having a trained model choose a tool for each. On 2,124 filtered ToolBench G3 test instructions, Qwen3-8B after SFT and RL achieves 41.5% exact-match success, versus 30.6% for prompted Llama-4-Maverick using a Search-R1-style protocol. This result is limited to tool selection on this benchmark.

Key insights

• Training both planning and selection matters: replacing the trained planner or selector with an untrained alternative substantially reduces exact-match success.
• The learned policy transfers across several tool domains and can benefit from a stronger retrieval backend without retraining the controller.
• Transfer is not uniform: on TaskBench, the 4B and 8B models score 72.8%, below the 80.6% Search-R1-style baseline, suggesting planning is especially sensitive to domain differences.

Summary by alphaXiv.