Reproduce “When AI output tips to bad but nobody notices: Legal implications of AI's mistakes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dylan J. Restrepo, Nicholas J. Restrepo, Frank Y. Huo, Neil F. Johnson
This research demonstrates that AI-generated legal fabrications are not random 'hallucinations' but result from a deterministic mechanism within the Transformer architecture, where an AI's internal state crosses calculable "tipping points" from reliable reasoning to authoritative falsehoods. Applying a physics-based model to a simulated legal case reveals how anomalous queries can destabilize the AI, making fabrication a foreseeable engineering risk with implications for professional responsibility.
Key insights
- AI fabrication results from deterministic mechanisms within the Transformer architecture, involving calculable "tipping points" where an AI's output transitions from reliable reasoning to authoritative falsehood.
- Anomalous legal queries serve as the primary destabilizing element, making AI most susceptible to fabrication precisely when human expertise and reliable information are critically needed.
- A sustained period of initially correct AI output paradoxically amplifies the risk of harm, as it fosters user trust, potentially leading to relaxed scrutiny before the AI crosses its fabrication threshold.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.