Reproduce “Attention to the Burstiness in Visual Prompt Tuning!” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuzhu Wang, Manni Duan, Shu Kong
Bilinear Prompt Tuning (BPT) is introduced to improve Visual Prompt Tuning (VPT) by addressing identified burstiness and non-Gaussian data distributions within the Transformer's self-attention module. The method consistently outperforms existing PEFT techniques across various vision tasks and model scales, achieving state-of-the-art accuracy while accelerating training and often reducing computational overhead.
Key insights
- Intermediate computations within the Transformer's self-attention module during prompt tuning exhibit pervasive "burstiness" and non-Gaussian (hyper-Laplacian and Laplacian) data distributions.
- Learning "bursty prompts" through bilinear operations, particularly after decorrelating non-Gaussian data, significantly enhances prompt tuning effectiveness and efficiency.
- The observed burstiness can be effectively mitigated or leveraged by transforming prompts to operate on more decorrelated and Gaussian-like data, or by designing prompts with inherent bilinear structures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.