Reproduce “Prefix Probing: Lightweight Harmful Content Detection for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jirui Yang, Hengqi Guo, Zhihui Lu, Yi Zhao, Yuansen Zhang, Shijing Hu, Qiang Duan, Yinggui Wang and 1 more
Prefix Probing presents a lightweight and accurate method for detecting harmful content in Large Language Models by leveraging an LLM's inherent generative biases and prefix caching. This black-box approach achieves detection performance comparable to dedicated external safety models with significantly reduced latency and no additional deployment cost.
Key insights
- LLMs exhibit inherent generative biases, assigning higher log-probabilities to 'refusal' prefixes for harmful inputs and 'agreement' prefixes for benign inputs.
- Standard LLM inference optimizations like KV prefix caching can be systematically leveraged for highly efficient and accurate black-box safety detection without modifying the model.
- Automated search-based construction of discriminative prefixes is more effective than manually designed prefixes for enhancing detection performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.