Reproduce “PLeak: Prompt Leaking Attacks against Large Language Model Applications” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, Yinzhi Cao

PLeak introduces an automated, closed-box attack framework for extracting confidential system prompts from Large Language Model (LLM) applications. This method, developed by researchers from Johns Hopkins University and Duke University, achieves an exact match accuracy of 68% against real-world applications on the Poe platform, significantly outperforming manual or adapted baseline techniques.

Key insights

  • Optimizing adversarial queries through an incremental search process, focusing on partial prompt reconstruction before extending to the full prompt, is essential for high-fidelity content extraction.
  • Aggregating information from multiple adversarial query responses significantly boosts the completeness and accuracy of system prompt reconstruction, even from fragmented leaks.
  • Adversarial transformations can allow prompt leaking attacks to bypass simple filter-based defenses, enabling the recovery of the original prompt's semantic content.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.