Reproduce “ParallelComp: Parallel Long-Context Compressor for Length Extrapolation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang and 2 more
ParallelComp introduces a training-free method for large language models to process ultra-long contexts by combining parallel chunking, novel KV cache eviction, and attention calibration. This approach enables 8B parameter models to extrapolate to 128K tokens on a single A100 GPU and achieves a 23.50x acceleration in the prefill stage.
Key insights
- Parallel attention exhibits distinct bias patterns (e.g., U-shape, Mountain-shape) that differ from classical attention, necessitating specialized mitigation strategies.
- Intelligent Key-Value (KV) cache eviction, by targeting specific high-score tokens, can effectively mitigate attention biases and lead to more uniform attention distributions.
- Chunk-based length extrapolation approaches generally yield superior performance compared to position-encoding-based methods for ultra-long context tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.