Reproduce “Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)” on OpenResearch

About this paper

Junki Mori, Kazuya Kakizaki, Taiki Miyagawa, Jun Sakuma

DP-SynRAG proposes a two-stage pipeline for generating a differentially private synthetic RAG database, addressing the privacy budget accumulation of query-time methods and the lack of locality preservation in prior synthetic text generation. This framework ensures a fixed privacy budget and retains fine-grained factual details, significantly improving RAG system scalability and utility in sensitive applications.

Key insights

• Decoupling privacy enforcement from query-time operations through a fixed-budget synthetic database generation significantly improves the scalability and long-term utility of privacy-preserving RAG.
• Differentially private soft clustering, combining keyword-based and embedding-based approaches, is crucial for preserving the essential "locality" and fine-grained factual information required by RAG systems.
• A post-processing self-filtering step, where an LLM evaluates the usefulness of generated synthetic texts for downstream tasks, can effectively enhance the overall utility of the synthetic RAG database.

Summary by alphaXiv.