Reproduce “Unified Deployment-Aware Evaluation of Open Reasoning Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Motaleb Hossen Manik, Ge Wang
A deployment-oriented evaluation of seven open reasoning-model configurations shows that benchmark accuracy alone does not determine the best practical choice. Across ARC-Challenge, GSM8K, MATH, and corrected TruthfulQA, Gemma-4-26B-A4B zero-shot reached the highest weighted score of 0.776, while Gemma-4-E4B few-shot CoT offered the strongest lower-memory option at 0.758 and about 14.9 GB VRAM.
Key insights
- Model selection is a deployment operating-point problem: the highest-accuracy configuration is not necessarily the best choice under memory or latency limits.
- Prompting affects both reasoning performance and whether outputs can be reliably scored; Phi-4-Reasoning’s very low strict scores were largely associated with missing or incompatible answer formats.
- Task specialization matters: Qwen3-8B few-shot CoT led GSM8K, while Gemma-4-26B-A4B zero-shot led the weighted aggregate, so no configuration dominated every benchmark.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.