Reproduce “Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, Mohit Iyyer

Researchers from UMass Amherst and Google demonstrated that paraphrasing significantly reduces the accuracy of current AI-generated text detectors. They proposed a retrieval-based defense that effectively identifies semantically preserved AI-generated content by comparing it against a database of known outputs, achieving high detection rates against such manipulated text.

Key insights

  • Current AI-generated text detectors are highly susceptible to paraphrase attacks, even when semantic meaning is preserved, rendering them largely ineffective against sophisticated adversaries.
  • A retrieval-based defense, which compares suspected text against a database of known AI generations, offers a robust and scalable method for detecting paraphrased AI content.
  • Large language models can be effectively fine-tuned to generate high-quality, controllable paraphrases of long-form text (DIPPER), capable of evading existing detection mechanisms.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.