Reproduce “On the Role of Summary Content Units in Text Summarization Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch and 8 more
This research from TU Darmstadt, University of Zurich, Google Research, and IBM Research Europe systematically investigates automated approaches for approximating Summary Content Units (SCUs) to make the Pyramid evaluation method more practical. The study demonstrates that large language model-generated units (SGUs) can closely approximate human-written SCUs, and reveals that simpler sentence-splitting baselines can be equally effective for certain evaluation scenarios, particularly for longer summaries or system-level comparisons.
Key insights
- LLM-generated SGUs (especially with GPT-4) achieved high approximation quality for human-written SCUs, demonstrating their potential for automation.
- Human-written SCUs consistently provided the highest correlation with human judgments, particularly for short, abstractive summaries.
- A simple sentence-splitting baseline performed surprisingly well, matching SCUs for system-level comparisons or evaluations of longer summaries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.