Reproduce “Understanding and Mitigating Role Spoofing in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ezra Edelman, Benjamin Edelman, Surbhi Goel, Eran Malach
The authors study role spoofing, in which language models treat attacker-written text in tool results as user instructions. They isolate misleading content from fake role structure, finding that models can rely on either cue and that fine-tuning against one attack type transfers poorly to the other. Linear probes detect tested role-spoofing attacks, but do not detect prompt injection generally.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.