Reproduce “Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad ElNokrashy, Badr AlKhamissi, Mona Diab
The authors introduce layer fusion to combine representations across a pretrained language model’s depth. Depth-Wise Attention (DWAtt) uses the final-layer representation to weight layer signals; Concat sums learned linear transforms. On CoNLL-03 named entity recognition, DWAtt’s dev-set F1 at 128-shot after 100 epochs exceeds Concat by 1.6 percentage points and RoBERTa-26 by 5.28, but Concat leads with the least data and short training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.