Reproduce “Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal” on OpenResearch

About this paper

Aojie Yuan, Zhiyuan Julian Su, Haiyue Zhang, Yi Nian, Yue Zhao

Researchers at the University of Southern California discovered that large language models (LLMs) possess strong internal awareness of their own reasoning errors in Chain-of-Thought (CoT) processes, often detectable from the first step and consistently outperforming textual confidence. However, these internal error signals are diagnostic rather than causal, meaning they indicate a problem but cannot be directly manipulated to improve or correct the model's reasoning performance.

Key insights

• LLMs exhibit robust internal hidden error awareness, with linear probes on hidden states achieving significantly higher accuracy (up to 0.977 AUROC) in detecting CoT errors than verbalized confidence or other baselines.
• There is a substantial "concealment gap" where crucial error information present in hidden states is not reflected in the generated text, leading to models expressing high confidence in unfaithful, incorrect reasoning.
• The internal error signals are diagnostic, not causal; direct interventions on these signals (e.g., steering, patching) did not causally improve or prevent errors in reasoning and sometimes degraded performance.

Summary by alphaXiv.