Reproduce “Quantized Reasoning Models Think They Need to Think Longer, but They Do Not” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sanae Lotfi, Polina Kirichenko, Steven Li, Zechun Liu
The authors test a training-free decoding penalty for overthinking amplified by post-training quantization; it lowers scores for 50 hesitation or redirection tokens. On MATH-500, Qwen-1.5B with 3-bit AWQ improves accuracy, the percentage of questions answered correctly, from 47.0% to 61.2%, while mean CoT length, average generated reasoning tokens, falls from 23.4K to 12.9K. Across tests, length falls 12–23% on average, but accuracy sometimes declines.
Key insights
- Aggressive PTQ does not merely degrade reasoning capacity but significantly exacerbates an existing tendency for "overthinking," causing models to reach correct intermediate answers but fail to commit, leading to greatly extended and often incorrect Chains-of-Thought.
- Quantization noise disproportionately perturbs next-token probability distributions at decoding positions where the full-precision model is already uncertain (high entropy), making these susceptible to sampling tokens associated with hesitation or branching thought processes.
- Targeting and penalizing specific "overthinking marker" tokens at inference time can effectively mitigate quantization-induced overthinking, suggesting that the underlying reasoning ability might be more robust than its expression in quantized models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.