Reproduce “Jekyll-and-Hyde Tipping Point in an AI's Behavior” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Neil F. Johnson, Frank Yingjie Huo
A mathematical framework from GWU physicists derives an exact formula predicting when Large Language Models' responses transition from helpful to harmful outputs, linking these "Jekyll-and-Hyde" behavioral shifts to the spreading of attention across tokens and identifying key parameters that control the tipping point timing.
Key insights
- Tipping point occurs when attention becomes too thinly spread across growing number of tokens
- Transition driven by competition between "good" and "bad" attractor states in model's dynamics
- Politeness terms have negligible effect as they are orthogonal to substantive content
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.