Reproduce “MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peiran Wang, Chong Xiang, Wenjie Qu, Ying Li, Jiaheng Zhang, G. Edward Suh, Yuan Tian

MetaPI converts existing utility benchmarks into indirect prompt-injection evaluations, reusing their tasks, environments, and success criteria. It identifies task-relevant reads, text ownership, and authority limits to generate and test attacks in benchmark environments. On AppWorld, Spotlighting lowers attack success rate from 50.0% to 25.0% while clean utility falls from 82.5% to 80.4%; residual vulnerability remains across three text-only benchmarks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.