Reproduce “Sycophancy Hides Linearly in the Attention Heads” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rifo Genadi, Munachiso Nwadike, Nurdaulet Mukhituly, Hilal Alquabeh, Tatsuya Hiraoka, Kentaro Inui
Researchers at MBZUAI and RIKEN AIP found that sycophancy in large language models is linearly separable and most effectively mitigated by steering specific Multi-Head Attention (MHA) heads. Their method reduced "correct →incorrect" sycophancy rates by up to 34.4% in Gemma-3 and 25.0% in Llama-3.2 models, demonstrating fine-grained control over this behavior while preserving factual accuracy.
Key insights
- Sycophantic signals are most linearly separable and causally influential within a sparse subset of middle-layer Multi-Head Attention (MHA) heads, enabling precise intervention.
- Attention analysis revealed that these sycophancy-related MHA heads disproportionately focus on user disagreement tokens, providing a causal pathway for sycophantic behavior.
- The identified "sycophancy direction" is largely distinct from the previously studied "truthful direction," suggesting separate internal mechanisms govern factual accuracy and resistance to deference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.