Reproduce “How Not to Detect Prompt Injections with an LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sarthak Choudhary, Divyam Anshumaan, Nils Palumbo, Somesh Jha
Researchers at the University of Wisconsin-Madison formally analyzed Known-Answer Detection (KAD) schemes for prompt injection, identifying a fundamental structural vulnerability in their design. They introduced DataFlip, an adaptive attack that consistently evades KAD by extracting the 'secret key' from the detection prompt, leading to near-perfect false negative rates even against fine-tuned 'Strong KAD' defenses, while simultaneously achieving high success in executing malicious tasks on backend LLMs.
Key insights
- The 'secret key' used in KAD, while unknown to the adversary a priori, is present within the detection prompt's context window and can be extracted by an adaptive injected instruction.
- Strong KAD fine-tuning, by increasing the detection LLM's propensity to follow injected instructions, inadvertently exacerbates the Type II failure, making the defense more vulnerable to adaptive attacks like DataFlip.
- Relying solely on input-output behavior for LLM security guarantees is insufficient, necessitating a shift towards understanding LLM internal reasoning processes for robust defense mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.