Reproduce “Enhancing Faithfulness in Abstractive Summarization via Span-Level Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sicong Huang, Qianqi Yan, Shengze Wang, Ian Lane
Researchers from the University of California, Santa Cruz introduced a methodology for enhancing the factual consistency of abstractive summaries by fine-tuning large language models using span-level hallucination annotations. Their approach leveraged GPT-4o to identify unfaithful text segments and demonstrated that Unlikelihood Training, in particular, robustly improved faithfulness across various models and datasets.
Key insights
- Span-level hallucination annotations, even when generated by an advanced LLM like GPT-4o, provide effective and granular feedback for fine-tuning models to improve factual consistency.
- Unlikelihood Training proved to be the most stable and robust fine-tuning method, consistently improving faithfulness across a wide range of negative sample weights (ϵ) without degrading model coherence.
- The BARTScore metric showed inconsistent and often contradictory results for faithfulness evaluation, suggesting its limitations when models are fine-tuned to specifically minimize the probability of certain tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.