Reproduce “Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization” on OpenResearch

About this paper

Siyao Li, Deren Lei, Pengda Qin, William Yang Wang

This research introduces a Distributional Semantic Reward (DSR) based on BERTSCORE for deep reinforcement learning in abstractive summarization, leading to generated summaries that are more semantically accurate, lexically diverse, and less repetitive. Models trained with DSR achieved higher human preference for relevance and fluency compared to baselines using traditional ROUGE-L rewards.

Key insights

• Optimizing directly for distributional semantic similarity (BERTSCORE) through RL effectively guides abstractive summarization models to produce outputs that are semantically richer and more aligned with human judgment than purely lexical matching (ROUGE-L).
• A pure DSR objective, without the additional cross-entropy loss (teacher forcing), can achieve comparable or even better human evaluation results for relevance and fluency, suggesting reduced reliance on traditional XENT for quality.
• DSR intrinsically encourages greater lexical diversity and significantly reduces repetition in generated summaries by moving beyond surface-level n-gram overlaps, addressing a common shortcoming of ROUGE-optimized systems.

Summary by alphaXiv.