Reproduce “ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyu Wang, Christopher M. Poskitt, Jiali Wei, Jun Sun
ProbGuard offers a proactive runtime monitoring framework for LLM agents, predicting safety violations probabilistically through learned Discrete-Time Markov Chains (DTMCs). It delivered advance warning times up to 38.66 seconds in autonomous driving and reduced unsafe behaviors by as much as 93.60% in embodied agents, all while maintaining minimal runtime overhead.
Key insights
- Proactive runtime monitoring, based on probabilistic risk prediction, can significantly enhance LLM agent safety by enabling timely interventions before violations fully materialize.
- Formal methods, specifically DTMCs and PAC analysis, can be effectively adapted to model and provide statistical guarantees for the stochastic and opaque behavior of LLM agents.
- Abstracting concrete system states into a finite set of symbolic states allows for learning tractable probabilistic models that capture complex agent-environment interactions relevant to safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.