Reproduce “The Model's Tell: Measuring Context-Leakage Attack Signals with Behavior Gauges” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maosen Zhang, Jianshuo Dong, Boting Lu, Wenyue Li, Xiaoping Zhang, Tianwei Zhang, Jie Zhang, Han Qiu
LeakGauge estimates leakage risk before generation by appending a fixed disclosure sentence and scoring its prefill token probabilities with a small probe. Across eleven models and system-prompt and RAG leakage tasks, its Behavior gauge achieves AUROC of at least 0.944 across evaluated splits. These scores indicate risk, not guaranteed leakage; tests show a known fixed probe can be targeted.
Key insights
- LLMs inherently expose a detectable 'tell' of leakage intent in their prefill token probabilities, accessible pre-decoding without deep architectural introspection.
- The content-agnostic 'Leakage Behavior' gauge captures a more abstract, robust disclosure signal compared to the content-specific 'Exact Content' gauge, generalizing better across diverse contexts and objectives.
- The observable attack-risk score derived from prefill probabilities is sensitive to the model's internal leakage-related representational direction, as evidenced by activation steering experiments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.