Reproduce “Adversarial Attacks against Neural Ranking Models via In-Context Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amin Bigdeli, Negar Arabzadeh, Ebrahim Bagheri, Charles L. A. Clarke
A novel black-box adversarial attack framework, Few-Shot Adversarial Prompting (FSAP), leverages Large Language Models to autonomously generate new, misleading documents designed to outrank credible content in neural ranking models. The method, particularly its FSAP_InterQ variant, achieved Mean Help-Defeat Rates up to 97.2% while maintaining high undetectability, posing a scalable threat to information retrieval systems.
Key insights
- LLMs can be effectively conditioned via few-shot prompting to autonomously generate novel, highly effective adversarial documents that deceive state-of-the-art Neural Ranking Models.
- The generated adversarial content exhibits a critical balance of high attack effectiveness, strong stance alignment, and significant undetectability by advanced LLM-based detectors.
- A small support set (around 5 examples) is sufficient for FSAP to achieve near-optimal adversarial performance, demonstrating the method's efficiency and scalability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.