Reproduce “Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering” on OpenResearch

About this paper

Michael J. Zellinger, Matt Thomson

Researchers at Caltech devised human-in-the-loop system architectures to enhance the reliability and efficiency of reasoning Large Language Models. Their "Fail Fast, or Ask" system reduced the conditional error rate for models like Qwen3 to less than 1% on difficult MATH problems and achieved up to 55% latency reduction and 64% cost savings for some configurations.

Key insights

• The length of a reasoning model's output trace serves as an effective proxy for its uncertainty in models like Qwen3 and DeepSeek R1, enabling reliable selective prediction.
• Integrating a faster, non-reasoning LLM can achieve substantial system-level latency and cost reductions, up to 55% and 64% respectively, while preserving high accuracy.
• The 'latency drag' phenomenon occurs when the non-reasoning model filters out easier queries, causing the reasoning model to process a disproportionately harder and slower conditional workload, thereby reducing expected efficiency gains.

Summary by alphaXiv.