Reproduce “SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference” on OpenResearch

About this paper

Abhijit Mishra, Mingda Li, Soham Deo

SentinelLMs introduces a lightweight, post-hoc adaptation and fine-tuning framework for transformer-based language models, enabling them to process passkey-encrypted inputs. This approach maintains performance parity with original models and prevents the recovery of sensitive user text from network interception or server-side logs.

Key insights

• Irreversible token-level encryption and spatial glide-reflection transformations on embeddings can secure input data while preserving semantic relationships crucial for model performance.
• Maintaining the relative spatial distances between embedding vectors, despite their absolute positions changing, enables adapted models to retain linguistic understanding.
• Combining key-based encryption with random shuffling of vocabulary and embedding indices significantly enhances robustness against reverse engineering attempts to recover original text.

Summary by alphaXiv.