Reproduce “When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoning” on OpenResearch

About this paper

Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary

This research introduces methods to quantify the faithfulness of latent reasoning in large language models solving mathematical problems, revealing that high benchmark accuracy frequently conceals computationally unstable processes. It found that 81.6% of correct answers emerged from unreliable pathways and identified an 8.8% silent failure rate, prompting a re-evaluation of LLM assessment beyond mere correctness.

Key insights

• LLM benchmark accuracy often masks a significant "fidelity gap," with a large proportion of correct answers arising from computationally inconsistent "Lucky Guesses" rather than stable, faithful reasoning.
• A notable "silent failure rate" exists, where models produce confident but incorrect predictions, posing a material safety risk for real-world deployment.
• Explicit Chain-of-Thought (CoT) prompting primarily improves performance by better aligning existing latent reasoning processes rather than fundamentally deepening the computation itself.

Summary by alphaXiv.