Reproduce “Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference” on OpenResearch

About this paper

Anton Xue, Avishree Khare, Rajeev Alur, Surbhi Goel, Eric Wong

Logicbreaks introduces a formal framework based on propositional Horn logic to analyze how large language models (LLMs) can be made to subvert prompt-specified rules. The study constructs theoretical transformer mechanisms for inference and rule subversion, then empirically demonstrates that these theoretically derived attacks and their underlying attention suppression mechanisms successfully transfer to both small learned reasoners and state-of-the-art LLMs like Llama-2 and Llama-3, enabling targeted rule disobedience.

Key insights

• Correct rule-following in propositional Horn logic can be formally characterized by Monotone, Maximal, and Sound (MMS) properties, providing a robust definition for desired behavior and its subversion.
• Rule subversion in transformers can be theoretically modeled as an exploitation of attention mechanisms' internal summation approximations, particularly through 'attention suppression' of targeted rules or facts.
• Adversarial prompts can induce specific types of rule disobedience by manipulating internal computational processes within transformers, and these mechanisms are transferable from theoretical constructions to practical LLMs.

Summary by alphaXiv.