Reproduce “Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts” on OpenResearch

About this paper

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang and 4 more

Rainbow Teaming introduces a black-box, open-ended approach using quality-diversity search and LLMs to generate diverse and effective adversarial prompts for large language models. The method achieved over 90% attack success rates on state-of-the-art LLMs and, when used for fine-tuning, reduced attack success rate to 0.3-0.7% on held-out prompts without compromising general performance.

Key insights

• Quality-Diversity search, specifically MAP-Elites, combined with LLM-based mutation and evaluation, can effectively generate both diverse and highly effective adversarial prompts in a black-box setting.
• Leveraging 'stepping stones' (mutating from existing successful prompts) and allowing cross-category mutations is crucial for exploring a wide and effective range of adversarial strategies.
• Preference-based evaluation by a Judge LLM is more robust against reward hacking and facilitates continuous exploration compared to score-based metrics, leading to more open-ended discovery.
• Synthetically generated, diverse adversarial data can significantly improve LLM safety through supervised fine-tuning without degrading general performance, supporting iterative self-improvement cycles.

Summary by alphaXiv.