Reproduce “Poolingformer: Long Document Modeling with Pooling Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, Weizhu Chen
Poolingformer presents a two-level attention mechanism that efficiently processes long documents by combining sliding window attention with a novel pooling attention. The model achieves new state-of-the-art performance on long-sequence Question Answering and Abstractive Summarization tasks, including an F1 score of 79.8 on Natural Questions long answer, while maintaining linear computational complexity.
Key insights
- A two-level attention mechanism efficiently extends the receptive field for long documents by combining local sliding windows with pooled, coarse-grained global context.
- Strategic application of pooling operations on keys and values in the second attention level effectively compresses information, enabling linear complexity even with broad contextual access.
- Selective integration of the Poolingformer attention into later layers of pretrained models maintains compatibility and leverages existing knowledge, preventing performance degradation from full layer replacement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.