Reproduce “Control Illusion: The Failure of Instruction Hierarchies in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann
Large Language Models struggle to reliably enforce instruction priorities when faced with conflicting directives, particularly within the widely adopted system/user message separation. The study finds that implicitly learned societal hierarchies can exert a stronger influence on model behavior than explicitly engineered instructional roles.
Key insights
- Explicit instruction hierarchies, such as system/user separation, often fail to provide reliable control, demonstrating a pervasive 'control illusion' in LLMs.
- LLMs rarely explicitly acknowledge conflicting instructions, and even when they do, it does not reliably lead to correct prioritization.
- Implicitly learned societal hierarchies (e.g., authority, expertise, social consensus) can exert a stronger influence on model behavior than explicitly engineered instructional roles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.