Reproduce “DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Zhao, Zuchao Li, Hai Zhao, Baoyuan Qi, Guoming Liu
A new method called DAC is introduced for task-agnostic prompt compression, addressing the limitations of prior entropy-based approaches by integrating attention scores and employing a dynamic, iterative compression procedure. It achieves an average F1 score of 37.76 on LongBench at 50% compression, representing a 4.03-point improvement over the previous best entropy-based method, LLMLingua.
Key insights
- Information entropy and accumulated attention scores are poorly correlated (Pearson 0.095), indicating that attention-critical tokens are not always high-entropy, thus requiring an integrated compression metric.
- Token information entropy is not static but dynamically changes during compression, especially when preceding tokens are removed, necessitating an adaptive compression strategy.
- Assigning a higher weight to accumulated attention scores (e.g., ".8" in additive fusion) within the integrated metric significantly improves prompt compression performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.