Reproduce “Language Model Inversion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, Alexander M. Rush
Researchers at Cornell University demonstrated that input prompts given to a language model can be effectively reconstructed by analyzing only the model's next-token probability distributions. The work achieved a BLEU score of 59 and exactly recovered 27% of prompts on Llama-2 7B models, highlighting a privacy vulnerability where sensitive user data could be leaked through language model APIs.
Key insights
- Language models encode significant information about previous tokens in their internal states, making prompt reconstruction from next-token probabilities feasible.
- Reinforcement Learning from Human Feedback (RLHF), a common safety measure, does not provide substantial protection against language model inversion attacks.
- Introducing randomness through temperature sampling during text generation can significantly reduce the effectiveness of prompt inversion.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.