Reproduce “DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, Chelsea Finn
DetectGPT introduces a zero-shot method for detecting machine-generated text by leveraging the distinct "probability curvature" of large language model outputs compared to human text. The approach, developed by Stanford University researchers, consistently outperforms other zero-shot baselines across diverse datasets and LLMs, including those with 175 billion parameters, demonstrating robustness to domain and language shifts.
Key insights
- Large language models generate text in regions of their probability space where the log-probability function exhibits negative curvature, appearing as "peaks" relative to nearby meaningful variations.
- This "probability curvature" offers an inherent and distinct statistical fingerprint for LLM-generated content, allowing for detection without explicit training or watermarking.
- The perturbation discrepancy metric directly relates to the negative trace of the Hessian of the log probability function, providing a theoretical foundation for the observed detectability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.