Reproduce “TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng
TriAttention combines attention skipping with mixed precision for long-document question answering, assigning query–key tiles to 8-bit or 4-bit computation, or skipping them, using calibrated per-head boundaries. On Llama-3.2-3B-Instruct, it scores 47.94 F1 on 64K LooGLE single-hop QA versus BF16’s 48.40, and reaches 22.80 tokens per second versus 14.06 on an A100 for a specified 65,536-token request; these results are configuration-specific.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.