Reproduce “Vision-Language Model IP Protection via Prompt-based Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lianyu Wang, Meng Wang, Huazhu Fu, Daoqiang Zhang

IP-CLIP introduces a lightweight, prompt-based learning strategy for Vision-Language Models (VLMs) like CLIP, enabling robust intellectual property protection by preserving high performance in authorized domains while deliberately degrading it in unauthorized ones. This method achieved the highest weighted drop scores on benchmarks like Office-31 (74.84%) and maintained a 0.0% degradation in authorized domain performance.

Key insights

  • Prompt-based learning offers an efficient and parameter-economical way to implement IP protection for large VLMs, avoiding the high computational costs of full model fine-tuning.
  • Integrating both image style and content information via learned prompts and a style-enhancement branch provides a semantically richer and more robust mechanism for distinguishing between authorized and unauthorized VLM usage.
  • Carefully constructed adversarial loss functions can effectively enforce domain-specific usage authorization, ensuring high performance only in intended domains and deliberate degradation elsewhere.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.