Reproduce “Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern” on OpenResearch

About this paper

Hongyin Tang, Di Xiu, Lanrui Wang, Xiurui Geng, Jingang Wang, Xunliang Cai

Ltri-LLM is a training-free framework for streaming long-context inference in Large Language Models that improves retrieval accuracy and reduces Key-Value (KV) cache memory usage by leveraging dynamic triangular attention patterns. It achieves superior performance on long-context benchmarks like ∞-Bench and RULER, while reducing KV cache GPU memory for 100K contexts to 0.027 GiB, orders of magnitude less than prior methods.

Key insights

• Large Language Models (LLMs) inherently exhibit strong local correlations in their attention distributions, forming 'triangular regions' that correspond to semantic segments or spans within the input.
• Existing streaming inference methods, like InfLLM, face a critical retrieval bottleneck, struggling to recall crucial tokens (even those with low attention scores) from long contexts, which impacts performance in tasks like Needle-In-A-Haystack.
• Dynamically adapting the number of index vectors per semantic span based on 'voter confidence' (ratio of a span's attention to its neighbors) can enhance retrieval accuracy while maintaining compression.

Summary by alphaXiv.