Reproduce “A Closer Look at System Prompt Robustness” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Norman Mu, Jonathan Lu, Michael Lavery, David Wagner
Researchers at UC Berkeley systematically evaluate and enhance the robustness of LLM system prompt adherence, demonstrating that even advanced models struggle with increasing guardrail complexity, but that fine-tuning with realistic data and Direct Preference Optimization (DPO) can enable smaller models to surpass larger proprietary ones in adherence. The work highlights a "Prompt Complexity Wall" where adherence drops significantly with more instructions.
Key insights
- LLMs face a "Prompt Complexity Wall," where adherence to system instructions rapidly declines as the number of guardrails increases, indicating a fundamental limitation in processing complex commands.
- Combining high-quality, diverse supervised fine-tuning data with Direct Preference Optimization (DPO) is highly effective for improving system prompt robustness, allowing smaller open models to achieve or exceed the performance of larger proprietary models.
- Inference-time methods like Classifier-Free Guidance consistently offer additional improvements in system prompt adherence, suggesting ways to amplify system instructions even without extensive re-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.