Reproduce “Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection” on OpenResearch

About this paper

Meng Chen, Kun Wang, Li Lu, Jiaheng Zhang, Tianwei Zhang

AudioHijack studies third-party attacks that alter supplied audio to steer assistants toward attacker-chosen instructions while leaving user requests unchanged. It optimizes subtle audio perturbations across contexts. Across 13 models, non-tool PISR averages 0.89–0.95 and BMSR 0.84–0.94; SpeechGPT scores lower. On Voxtral-Mini, context-agnostic attacks achieved BMSR above 0.77, versus 0.18–0.65 for context-specific attacks on unseen contexts. The method requires model architecture and parameter access.

Key insights

• LALMs are fundamentally vulnerable to third-party auditory prompt injection across various architectures, scales, and even commercial systems, with over-optimization for structured outputs increasing susceptibility to tool misuse.
• AudioHijack effectively shifts the LALM's attention from user context to the adversarial audio, enabling robust generalization to unseen user instructions.
• Existing defense mechanisms (prompt-level, response-level, or logit-level) are largely ineffective against these attacks, highlighting the need for deeper internal scrutiny, such as attention deviation detection, though an effectiveness-detectability trade-off exists for adaptive attackers.

Summary by alphaXiv.