Reproduce “PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiyuan Zhang, Chuan Wang, Joey Zhong, Paul Fryzel, Kyle Polley, Jerry Ma, Ninghui Li
PII-Tracer evaluates repeated private identifiers across conversation turns using a compact encoder trained to detect linked mentions in context. On a synthetic benchmark of 13,148 dialogues in 13 languages, it achieves a 0.776 cross-turn consistent-detection rate, versus 0.551 for GPT-5.6-sol, but flags 0.385 of PII-free documents. These results show improved mention-chain coverage alongside false alarms, not production safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.