Reproduce “A Neural Attention Model for Abstractive Sentence Summarization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander M. Rush, Sumit Chopra, Jason Weston

Researchers from Facebook AI Research and Harvard SEAS introduced a neural attention model for abstractive sentence summarization, adapting techniques from neural machine translation. The model, particularly with extractive tuning, surpassed previous statistical and linguistically-motivated baselines on DUC-2004 and Gigaword benchmarks, achieving a ROUGE-2 score of 8.49 on DUC-2004, which exceeded human inter-annotator agreement.

Key insights

  • Neural attention mechanisms, adapted from machine translation, are highly effective for abstractive summarization, significantly reducing perplexity compared to simpler encoders.
  • A hybrid approach combining a neural generative model with extractive tuning (ABS+) improves performance by leveraging both abstractive generation and strategic word/phrase transfer from the source.
  • End-to-end training of a structurally simple neural model on large datasets (4 million article-headline pairs) is feasible for abstractive summarization, enabling it to learn complex patterns without explicit linguistic rules.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.