Reproduce “Effective Code Membership Inference for Code Completion Models via Adversarial Prompts” on OpenResearch

About this paper

Yuan Jiang, Zehao Li, Shan Huang, Christoph Treude, Xiaohong Su, Tiantian Wang

AdvPrompt-MIA, a new framework for membership inference attacks on code completion models, leverages semantics-preserving adversarial prompts to amplify memorization signals. The approach constructs a deep learning classifier based on output stability and perplexity, achieving high accuracy (e.g., 0.95 AUC on APPS, 0.97 AUC on HumanEval) and strong generalizability across various code LLMs.

Key insights

• Memorized training samples elicit stable outputs from code LLMs even when subjected to small, semantics-preserving input perturbations, whereas non-member samples result in more variable outputs.
• The 'behavioral stability' gap, particularly the standard deviation of similarity scores across perturbed outputs, provides a crucial and fine-grained signal for membership inference in code LLMs.
• Structurally valid yet semantically inert code blocks, such as dead loops, are particularly effective in exposing subtle memorization discrepancies, offering unique insights into how LLMs process code.

Summary by alphaXiv.