Reproduce “Multi-Agent AI Safety as an Institutional Design Problem” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abdullah X
The authors study how institutions govern AI workflows by stating rules, tracking authority, enforcing actions, and handling blocks. In delegation tasks, a constitutional prompt and provenance-aware guard produced 0 realized violations in 384 episodes through prevention and containment, respectively. In 96 matched laundering episodes, provenance enforcement had 0 violations versus 22 for a local guard; results are limited to synthetic tasks.
Key insights
- Achieving zero realized violations can stem from two distinct mechanisms: 'behavioral prevention,' where agents do not attempt prohibited actions, and 'mechanical containment,' where an external system blocks unsafe proposals, implying that final violation rate alone is an incomplete safety metric.
- Enforcement mechanisms that rely on mutable visible policy are vulnerable to 'transformation-laundering' where ordinary data changes bypass restrictions, highlighting the importance of provenance-aware enforcement that tracks immutable originating authority.
- Institutional design can influence agent behavior through informational channels, as demonstrating numerical limits can inadvertently create focal points that alter agent requests even without changing mechanical constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.