Reproduce “On Learning to Summarize with Large Language Models as References” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan

Smaller text summarization models can achieve significantly improved performance by training on higher-quality reference summaries generated by Large Language Models (LLMs). The work also critically examines LLM-based evaluation methods, revealing their limitations and biases when comparing closely performing systems.

Key insights

  • Using LLM-generated references for training substantially improves the quality of smaller summarization models, enabling them to distill LLM capabilities and achieve comparable performance to LLMs under LLM-based evaluation.
  • Contrastive learning consistently outperforms standard MLE fine-tuning when training with LLM-generated references or LLM-based evaluations, optimizing models more effectively towards higher quality outputs.
  • While LLM-based evaluation methods show strong performance on general benchmarks, they demonstrate limited alignment with expert human judgments and exhibit biases (e.g., positional bias, self-inconsistency) when comparing closely performing summarization systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.