Reproduce “Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classification” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Muhammad ElNokrashy, Badr AlKhamissi, Mona Diab

The authors introduce layer fusion to combine representations across a pretrained language model’s depth. Depth-Wise Attention (DWAtt) uses the final-layer representation to weight layer signals; Concat sums learned linear transforms. On CoNLL-03 named entity recognition, DWAtt’s dev-set F1 at 128-shot after 100 epochs exceeds Concat by 1.6 percentage points and RoBERTa-26 by 5.28, but Concat leads with the least data and short training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.