Reproduce “Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuyang Dai, Yushun Dong

Researchers at Florida State University introduced "Knowledge Trap," a defense mechanism designed to mitigate large language model (LLM) extraction attacks. It strategically redirects an attacker's query budget towards low-utility knowledge through a Honeypot Knowledge Graph and a breadcrumb luring system, resulting in a 6.2% average reduction in surrogate model agreement without impacting legitimate user experience.

Key insights

  • Model extraction attacks can be mitigated by manipulating the attacker's knowledge acquisition trajectory, guiding them to expend query budget on low-value information.
  • Creating a 'honeypot' of domain-adjacent but task-irrelevant knowledge (HKG) can effectively absorb an attacker's limited queries while maintaining the utility of the victim model for benign users.
  • A combination of early detection of malicious query patterns and adaptive breadcrumb injection can effectively steer active-learning-based attackers into a self-reinforcing loop within the honeypot knowledge space.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.