Reproduce “PRWeaver: Evaluating LLM-Based Code Auditors against Long-Horizon Malicious Pull Requests” on OpenResearch

About this paper

Yuekun Wang, Mingfei Cheng, Xiaofei Xie

Singapore Management University researchers developed PRWeaver, a benchmark evaluating LLM-based code auditors against long-horizon malicious pull requests. The study found that such attacks, especially when disguised with benign changes, significantly reduce detection rates, with the strongest auditor system exhibiting an evasion rate of 31-34% for interleaved or carrier-fused attacks.

Key insights

• The way information is actively presented to an LLM auditor in its review context causes more significant detection degradation than merely having that information available in repository history.
• LLM auditors can localize malicious code but often fail to correctly characterize its security consequences or recover the combined security impact, indicating a 'deliberative' failure mode.
• Sophisticated attack presentations, specifically coherent carrier fusion and benign interleaving, are more effective at evading detection than simply decomposing an attack across multiple commit boundaries.

Summary by alphaXiv.