Reproduce “On Positional Bias of Faithfulness for Long-form Summarization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

David Wan, Jesse Vig, Mohit Bansal, Shafiq Joty

An investigation by researchers from UNC Chapel Hill and Salesforce AI Research systematically analyzes how positional bias affects the factual consistency of summaries produced by large language models for long documents. The study identifies a "U-shaped" trend in faithfulness, where LLMs are less accurate with information located in the middle of long inputs, and proposes an improved evaluation methodology for long-form summarization metrics.

Key insights

  • Positional bias is present not only in LLM generation but also in LLM-based faithfulness evaluation metrics, requiring specific strategies to mitigate metric-level bias.
  • Large Language Models consistently produce less faithful summaries for information located in the middle of long input documents, exhibiting a "U-shaped" faithfulness pattern.
  • While basic prompting can influence focus, complex generation strategies like hierarchical merging or incremental updating are generally ineffective at improving overall faithfulness or addressing positional bias, sometimes introducing new issues.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.