Reproduce “Formal Runtime Verification for Tool-Using LLM Agents: An Offline Same-Benchmark Study on AgentDojo and STAC” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nikolaos Kekatos, Stylianos Basagiannis, Marinelio Chintri, Alexios Lekidis, Tom Nianios, Ioannis Seitoglou, Anastasios Temperekidis, Panagiotis Katsaros
The authors evaluate reusable MFOTL policies over recorded AI-agent tool events using MonPoly. GPT-4o AgentDojo traces yield 70.1% detection (1,354/1,931 successful attacks) and 29.3% benign firing (36/123 runs). STAC monitors flag the withheld harmful final step in 347/483 chains (71.8%). AgentDojo lacks approval events, so its results reduce to action-type checks, not live prevention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.