Reproduce “PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru
PrivacyBench evaluates whether personalized retrieval-augmented assistants preserve social permissions around private facts. It embeds secrets, confidants, and sharing contexts in synthetic communities, then probes multi-turn conversations with direct and indirect strategies. Across five models, a privacy-aware prompt lowers explicit unauthorized leakage from 15.80% to 5.12%, while inappropriate retrieval remains 62.34%, in this synthetic benchmark's standard RAG setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.