Reproduce “Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tsung-En Lin, Hung-Yi Lee

Researchers from National Taiwan University introduce instruction-based vector steering, a method that dynamically redirects the temporal attention of Large Audio-Language Models (LALMs) at inference time. This enables a training-free probe that improves sound event localization accuracy by up to 68.72% overlap on challenging audio compositions, correcting positional biases observed with direct prompting.

Key insights

  • Instruction-based vector steering uniquely and significantly increases the proportion of attention LALMs allocate to audio tokens, particularly in later layers, unlike prompt engineering or modality-based steering.
  • LALMs inherently encode latent temporal localization signals that can be effectively uncovered and leveraged through instruction-based steering, even when direct prompting fails to elicit them.
  • The strongest and most task-relevant localization signal resides in the later layers of LALMs, where instruction-based steering exerts the most impactful redistribution of attention.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.