Reproduce “TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bohao Chu, Meijie Li, Sameh Frihat, Chengyu Gu, Georg Lodde, Elisabeth Livingstone, Norbert Fuhr
Researchers from the University of Duisburg-Essen and collaborators introduce TRACSUM, a new benchmark and evaluation framework for generating aspect-based summaries in the medical domain with sentence-level traceability. Their proposed TRACK-THEN-SUM pipeline and fine-tuned LLMs achieve competitive performance, establishing a reliable method for verifiable medical summarization.
Key insights
- Explicitly tracking relevant source sentences *before* generating a summary (TRACK-THEN-SUM) is more effective for ensuring traceability than end-to-end or summary-then-track approaches.
- Incorporating the full document context during the summarization phase significantly improves the completeness of generated aspect-based summaries without sacrificing other quality metrics.
- The choice of the NLI (Natural Language Inference) model used within the automatic evaluation framework critically impacts its correlation with human judgments, with larger LLMs (e.g., GPT-4o) showing stronger alignment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.