Reproduce “Adapters Strike Back” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jan-Martin O. Steitz, Stefan Roth
An in-depth investigation into adapter configurations for Vision Transformers (ViTs) reveals that an optimized adapter architecture, named Adapter+, achieves state-of-the-art performance on visual transfer learning benchmarks. The work demonstrates that careful design, including a novel learnable channel-wise scaling, overcomes previous underperformance and establishes adapters as a highly robust and parameter-efficient tuning method.
Key insights
- The 'Post-Adapter' position, where the adapter is inserted at the very end of a transformer layer after the Feed-Forward Network and its skip connection, consistently yielded the best performance.
- A newly proposed learnable channel-wise scaling mechanism applied to the adapter output is crucial for performance gains, contributing significantly more than scalar scaling and making internal Layer Normalization unnecessary.
- Previous reports of adapter underperformance in computer vision were largely attributable to sub-optimal implementations, such as incorrect adapter positions or missing critical components, rather than inherent limitations of the adapter concept.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.