Reproduce “SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wencheng Zhang, Shiqin Qiao, Lingjie Luo, Yinfeng Li, Chuanyang Zheng, Qian Xu, Meng Li, Yong Gui and 4 more

SynapseRoute, developed by Bytedance, Xidian University, CUHK, and Peking University researchers, presents an automated routing framework for dual-mode Large Language Models. It dynamically assigns medical queries to either a 'thinking' or 'non-thinking' inference mode, achieving 0.8390 overall accuracy while reducing inference time by 36.8% and token consumption by 39.66% compared to using the reasoning mode alone.

Key insights

  • Approximately 57.8% of medical questions can be accurately and efficiently answered by the less expensive 'non-thinking' LLM mode, indicating significant potential for cost savings through routing.
  • The 'thinking' (reasoning) mode of an LLM can exhibit 'overthinking' on simple tasks, introducing logical noise and potentially decreasing accuracy, highlighting the benefit of dynamic mode selection.
  • Explicitly classifying task complexity via a lightweight, dedicated classifier is effective for efficient resource allocation, outperforming direct LLM fine-tuning for this specific routing task.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.