Reproduce “Watermarking Pre-trained Language Models with Backdooring” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, Cho-Jui Hsieh
This paper presents a method for watermarking pre-trained language models (PLMs) in a black-box setting, allowing owners to verify their intellectual property after fine-tuning. The proposed Watermarking Pre-trained Language Model (WLM) approach enables multi-task watermarking that is robust to subsequent fine-tuning and utilizes stealthy, common-word combination triggers, achieving over 90% watermark extraction success rates across various NLP tasks.
Key insights
- Hard parameter sharing of backdoor trigger embeddings enables simultaneous watermarking across multiple distinct downstream tasks.
- Watermarks embedded with the WLM method exhibit high transferability across different task types (e.g., from topic classification to sentiment analysis), an unexpected beneficial property.
- Using combinations of common words as triggers significantly enhances watermark stealthiness, making them harder to detect while maintaining high extraction success rates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.