Reproduce “Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang
The authors test whether classifier probabilities can detect and stop prompt-injection hijacks in English four-label classification. Low confidence ranks hijacks above clean answers on four of five systems on AG News, but a gate calibrated to a 5% clean-alarm target misses 67% of gpt-6-luna’s plain-note hijacks and 99% of certain-note hijacks. These findings are limited to the tested systems and prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.