Reproduce “Inference-Time Attention Steering for Vision-Language-Action Driving Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Darshan Nagendra Prasad, Lars Ullrich, Knut Graichen
Researchers at FAU Erlangen-Nürnberg developed an inference-time attention steering mechanism for Vision-Language-Action (VLA) driving models, influencing ego-vehicle trajectory generation by biasing visual attention towards specific traffic actors. The method demonstrated a monotonic dose-response relationship, increasing Average Displacement Error (ADE) up to 17 cm mean for a `head_scale` of 4, while revealing that the model's Chain-of-Causation reasoning pathway remained unperturbed due to an exposure artifact.
Key insights
- Inference-time visual attention steering predictably influences a VLA model's trajectory output, with the effect predominantly localized and accumulating within later transformer layers.
- The Chain-of-Causation (CoC) textual reasoning was consistently invariant to the intervention, not due to architectural robustness, but because a per-call audit revealed the mask-based bias never reached the CoC generation pathways.
- This verified exposure asymmetry suggests potential for decoupled control—modulating motion planning via trajectory output without altering the transparent reasoning chain—but implies the CoC cannot serve as a reliable monitor for such interventions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.