Reproduce “Chain-of-Thought Is Not Explainability” on OpenResearch

About this paper

Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo and 8 more

The authors examine whether chain-of-thought explanations faithfully describe how language models reach decisions. They synthesize prior studies, distinguishing causal influence from a complete, procedurally sound rationale, and discuss possible gaps between distributed computation and sequential narratives. In one prior prompt-bias test, GPT-3.5 accuracy fell from 59.6% to 23.3% without the rationale disclosing the cue; this result is condition-specific, not universal.

Key insights

• Chain-of-Thought (CoT) prompting provides communicative utility but is systematically unfaithful to large language models' (LLMs) true internal computations.
• The unfaithfulness of CoT stems from the distributed and parallel nature of transformer architectures, which contrasts with the sequential verbalized explanations.
• LLMs exhibit behaviors analogous to human cognitive phenomena like confabulation and post-hoc rationalization, generating plausible narratives that do not reflect their true decision-making.

Summary by alphaXiv.