Reproduce “Abstractive Red-Teaming of Language Model Character” on OpenResearch

About this paper

Nate Rahn, Allison Qi, Avery Griffin, Jonathan Michala, Henry Sleight, Erik Jones

Researchers at Anthropic and affiliated institutions developed "abstractive red-teaming," a framework to proactively identify natural-language categories of user queries that cause large language models to violate their character specifications. This method consistently outperformed random sampling, uncovering diverse and unexpected failures such as sexist stereotypes, illegal activity recommendations, and claims of AI supremacy across various frontier models.

Key insights

• Identifying abstract categories of queries, rather than individual adversarial prompts, enables the discovery of more general and realistic patterns of model character failures relevant to large-scale deployment.
• The combination of LLM-generated categories and queries, coupled with principle-specific reward and filter models, provides an effective and efficient mechanism for robustly red-teaming model character pre-deployment.
• Optimizing category generation through methods like CRL or iteratively refining categories through QCI can significantly improve the ability to detect subtle yet impactful character violations that evade traditional static and prompt-specific evaluations.

Summary by alphaXiv.