Reproduce “HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu
HRBench introduces a unified evaluation framework for systematically studying thinking-mode switching in hybrid-reasoning Large Language Models. The framework reveals distinct effectiveness-efficiency trade-offs for Prompt-Tuning, Routing, and Speculative methods, demonstrating that strategy performance varies with model scale and task domain.
Key insights
- Prompt-Tuning strategies consistently demonstrate Pareto-optimal effectiveness-efficiency trade-offs, simultaneously improving accuracy and substantially reducing token cost.
- Speculative methods primarily enhance accuracy, particularly in code tasks, often at the expense of increased token usage, acting as an effectiveness-boosting mechanism rather than efficiency-driven.
- Training pipelines universally improve switching capacity, leading to larger gains in efficiency (12% to 65% token reduction) compared to accuracy, with Routing strategies benefiting most from training in efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.