Reproduce “Pretraining Data Can Be Poisoned through Computational Propaganda” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo
Researchers from the University of Washington demonstrated that large language model pretraining data can be poisoned by injecting content into public discussion interfaces on the open web. Their HALFLIFE framework estimated an end-to-end poison inclusion probability of 0.13%, which was sufficient to induce measurable shifts in base model behavior, with a 1.3B model showing a +19.0% preference shift at a 0.1% token poison rate.
Key insights
- Web-scale poisoning of large language model pretraining data is a practical threat via third-party content injection, with public comments providing a substantial, viable attack surface.
- Current data curation pipelines, primarily designed for quality and deduplication, are insufficient to detect and filter out adversarially injected content, especially when it mimics natural language.
- While instruction tuning can reduce the poisoning effect, base models remain highly susceptible, embedding biases that persist to some degree post-fine-tuning, particularly for naturalistic poison formats.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.