Reproduce “Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Philippe Laban, Alexander R. Fabbri, Caiming Xiong, Chien-Sheng Wu

A new benchmark, SummHay, is introduced to evaluate long-context Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) systems by tasking them with query-focused summarization and accurate source attribution across a synthetic 100k-token document corpus. This framework demonstrates that leading models achieve a maximum joint score of 44.6%, substantially below human performance of 56.1%, highlighting current limitations in complex information synthesis and verifiable generation.

Key insights

  • SummHay proves to be a challenging benchmark, as current state-of-the-art systems achieve Joint Scores significantly lower than human performance (44.6% vs. 56.1%).
  • There is a clear trade-off between information coverage and precise citation, with effective retrieval in RAG systems dramatically improving citation quality over full-context LLMs.
  • All tested long-context LLMs exhibit position bias, performing differently depending on where relevant documents are placed within the input context (e.g., top, bottom, middle).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.