Reproduce “Breaking the 1.58-bit Barrier for Ternary LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Evangelos Georganas, Alexander Heinecke, Pradeep Dubey
BITCOS changes how existing ternary language-model weights are stored, using a presence bit for each weight and sign bits only for nonzero weights. Its symbols-only cost is 2 − z bits per weight, where z is zero density; 26 of 29 surveyed checkpoints beat five-trit packing. Tests show higher decode throughput on memory-bound platforms, but lower throughput on Lunar Lake CPUs.
Key insights
- The practical crossover with five-trit packing occurs at 37.5% zeros; 26 of the 29 evaluated ternary models exceed this density, while BITCOS is smaller than two-bit storage for all measured models.
- Compression benefits translate into speed only when memory traffic is the bottleneck. Decode overhead can dominate on systems with high bandwidth per core, as observed on Lunar Lake.
- The method is bit-exact and can be applied to existing ternary checkpoints without retraining or enforcing a new sparsity pattern.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.