Reproduce “Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer” on OpenResearch

About this paper

Simon Schrodi, Elias Kempf, Fazl Barez, Thomas Brox

Researchers elucidated the mechanisms behind 'subliminal learning' in large language models, showing that hidden biases transfer during distillation via rare 'divergence tokens' processed primarily in early model layers. This work demonstrated that such bias transfer persists even when mitigating logit leakage and token entanglement, but can be suppressed by prompt paraphrasing or mixing data from multiple teachers.

Key insights

• Subliminal learning persists even when statistical logit leakage and entangled tokens are explicitly mitigated, demonstrating these are not necessary conditions.
• Rare "divergence tokens" - where biased teachers diverge in their greedy predictions - are identified as the causal drivers for subliminal bias transfer.
• Early layers of the student model are disproportionately critical for acquiring and encoding these subliminal biases from divergence tokens.

Summary by alphaXiv.