Reproduce “Core Context Aware Transformers for Long Context Language Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaofo Chen, Zeng You, Shuhai Zhang, Haokun Li, Yirui Li, Yaowei Wang, Mingkui Tan
Core Context Aware (CCA) Attention is a plug-and-play module that dynamically identifies and prioritizes core context while preserving local information for long sequence language modeling. It achieves a 7.9x inference speedup and significantly reduces KV cache memory compared to full self-attention, while maintaining or improving performance on benchmarks like LongBench-E for context lengths up to 128K.
Key insights
- Dynamically identifying and prioritizing 'core context' can effectively reduce context redundancy and computational load without sacrificing crucial information.
- A combination of coarse-grained global attention (on core tokens) and fine-grained local attention provides a robust way to model long-range and immediate dependencies simultaneously.
- Designing efficient attention mechanisms as plug-and-play modules enables leveraging existing powerful pretrained LLMs, accelerating adoption and reducing retraining costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.