Reproduce “LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation” on OpenResearch

About this paper

Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, Danqi Chen

Princeton PLI and The University of Texas at Austin developed LONGPROC, a multi-task benchmark evaluating long-context language models on generating structured, long-form outputs from dispersed information. Their evaluation of 23 LCLMs revealed significant performance degradation for all models as output length increased, with the best-performing model achieving only 54% accuracy on 8K-token tasks, highlighting struggles with long-range coherence and complex reasoning.

Key insights

• LCLMs demonstrate significant limitations in long procedural generation, with all tested models experiencing substantial performance degradation as required output length increases, especially beyond 2K tokens.
• Model scale is a crucial factor, with larger models consistently outperforming smaller ones, and reasoning-trained models generally showing stronger performance but sometimes suffering from token inefficiency.
• LCLMs struggle with long-range coherence, showing clear performance degradation in later segments of generated text and difficulty maintaining logical consistency throughout the generation process.

Summary by alphaXiv.