Reproduce “Ghostbuster: Detecting Text Ghostwritten by Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vivek Verma, Eve Fleisig, Nicholas Tomlin, Dan Klein
The UC Berkeley team developed Ghostbuster, a system for detecting text ghostwritten by Large Language Models that operates without requiring access to the target model's internal probabilities. The system achieves a 99.0 F1 score in-domain and demonstrates robust generalization across different writing domains and prompting strategies.
Key insights
- AI-generated text can be reliably detected by analyzing statistical properties derived from "weaker" language models, even without access to the target generation model's internal probabilities.
- Combining features from multiple weaker language models through a structured search significantly improves generalization and detection performance compared to simpler or overly complex neural approaches.
- Human reviewers struggle to identify AI-generated text (average 59% accuracy), underscoring the necessity of automated detection tools.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.