Reproduce “Hidden in Plain Sight: Exploring Chat History Tampering in Interactive Language Models” on OpenResearch

About this paper

Cheng'an Wei, Yue Zhao, Yujia Gong, Kai Chen, Lu Xiang, Shenchen Zhu

Researchers from the Chinese Academy of Sciences demonstrated that interactive Large Language Models (LLMs) are susceptible to "chat history tampering," an attack where fake conversational history is injected to manipulate model behavior. Their LLM-Guided Genetic Algorithm (LLMGA) successfully discovered black-box injection templates, which increased the success rate of eliciting disallowed content by LLMs to up to 98% on various models, circumventing existing safety mechanisms.

Key insights

• LLMs interpret conversational context semantically rather than by strictly parsing exact ChatML syntax, making them vulnerable to varied token structures and enabling black-box tampering.
• Injecting spurious chat history significantly increases LLM malleability, capable of amplifying inherent biases and drastically boosting the success rates of eliciting disallowed content.
• The LLM-Guided Genetic Algorithm (LLMGA) is effective at discovering attack templates in a black-box setting, often outperforming or complementing existing prompt injection and jailbreaking methods.
• "Word anonymization," when combined with history tampering, is highly effective at bypassing safety filters, suggesting LLMs may over-rely on keyword detection for harmful content.

Summary by alphaXiv.