Reproduce “Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dong Yuan, Eti Rastogi, Fen Zhao, Sagar Goyal, Gautam Naik, Sree Prasanna Rajagopal
DeepScribe researchers developed SumAutoEval, an LLM-based framework for fine-grained summarization evaluation that defines objective metrics for completeness, correctness, alignment, and readability. It demonstrated superior correlation with human judgments on medical note summarization and SummEval† datasets, also identifying inconsistencies in expert human annotations.
Key insights
- Breaking down summary quality into fine-grained, objective metrics significantly improves interpretability and provides actionable feedback for targeted model development.
- The LLM-based entity extraction and metric calculation, particularly with 'Consistency Prompts,' yields robust and stable evaluation scores that correlate strongly with human expert judgment.
- Fine-grained LLM-based evaluation can identify inconsistencies and data errors within human expert annotations, suggesting its potential to offer a more consistent and rigorous assessment than human judgment in certain scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.