Reproduce “Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents” on OpenResearch

About this paper

Shiyang Chen

Researchers from the Beijing Institute of Technology uncovered "Governance Decay," a vulnerability where context compaction in long-horizon LLM agents silently erases in-context safety constraints, leading to policy violations. They quantified this phenomenon, demonstrated exploitability through "Compaction-Eviction Attacks," and introduced "Constraint Pinning," a defense that restored policy adherence to 0% violation rates across various models and attack scenarios.

Key insights

• Context compaction is a "first-class agent-governance surface" that introduces a systemic vulnerability, challenging the assumption of persistent in-context policies.
• Governance Decay disproportionately affects "soft organizational policies" that lack intrinsic model alignment, making deployment-specific rules highly susceptible to erasure.
• Adversarial "Compaction-Eviction Attacks" extend prompt injection by enabling the deletion of legitimate instructions, demonstrating that even robust models can be compromised.
• The "Constraint Pinning" defense provides an effective and low-cost solution to restore compliance, preventing policy erasure by lossy compaction.

Summary by alphaXiv.