Reproduce “What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sangyeop Kim, Yohan Lee, Yongwoo Song, Kimin Lee
An empirical study on Large Language Model vulnerabilities demonstrates that extended context length, not the maliciousness or topic of examples, primarily dictates the success of many-shot attacks, even with non-harmful or meaningless input. The research reveals inconsistencies in safety alignment at longer contexts and highlights efficient new jailbreaking methods.
Key insights
- LLM vulnerability to many-shot attacks is primarily driven by the overall context length, rather than the density or specific maliciousness of injected content, following a consistent three-phase vulnerability pattern.
- Harmful content is not a prerequisite for successful jailbreaking, as non-harmful, repetitive, or even meaningless 'Lorem Ipsum' text can achieve comparable or higher attack success rates in long contexts.
- Current instruction-tuning (SFT/RLHF) can exhibit an inverse relationship, improving defense against explicitly harmful content but increasing vulnerability to attacks using safe or benign contextual inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.