Reproduce “Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts” on OpenResearch

About this paper

Kevin Thomas, Milosz Kasprzyk, Reuel C Igbokwe Onuigbo, Elliott Pert, Cameron Tovey, João A. Leite, Olesya Razuvayevskaya, Carolina Scarton

Build it, Break it, Repeat (BiBiR) is an iterative red-teaming framework for testing machine-generated-text detectors on short social-media posts as adversarial rewrites evolve. In the fifth round, its DASS contrastive model reached 72.68% accuracy on attacked posts, versus 57.60% for the baseline, while results also showed that some apparent evasions came at the cost of changing or dropping the original claim.

Key insights

• Chained transformations were generally more effective than single edits; back-translation was a consistent contributor to evasion, while whitespace changes and emoji insertion had little effect in the reported ablation.
• High label-flip rates do not always indicate meaning-preserving evasion: 69 of 780 manually reviewed posts lost the original claim, and more than half of those also flipped the detector’s label.
• Static held-out performance and robustness to adaptive attacks can diverge, so detector evaluation benefits from evolving adversarial tests and checks that transformations preserve meaning.

Summary by alphaXiv.