Reproduce “Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models” on OpenResearch

About this paper

Sizai Hou, Songze Li, Baturalp Buyukates

This paper introduces SecFPP, a novel protocol for privacy-preserving prompt personalization in federated learning for multimodal large language models (MLLMs). SecFPP achieves state-of-the-art personalization performance comparable to non-private methods while providing information-theoretic privacy guarantees for user-specific prompts with minimal computational and communication overheads.

Key insights

• Cryptographic secret sharing (LCC) offers a robust alternative to Differential Privacy, providing information-theoretic privacy guarantees for prompts without the performance compromise observed in noise-based methods.
• Decomposing prompts hierarchically and combining it with privacy-preserving adaptive clustering allows for effective adaptation to multi-granular data heterogeneity while maintaining strong privacy for individual user customizations.
• Empirical analysis shows that the information leakage from shared distance metrics used for clustering is negligible, providing a rigorous information-theoretic bound against adversarial prompt reconstruction.

Summary by alphaXiv.