Reproduce “VcLLM: Video Codecs are Secretly Tensor Codecs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ceyu Xu, Yongji Wu, Xinyu Yang, Beidi Chen, Matthew Lentz, Danyang Zhuo, Lisa Wu Wills

Researchers from Duke University and Carnegie Mellon University introduced VcLLM, a framework that repurposes existing hardware video codecs as general-purpose tensor codecs for Large Language Models. This method achieves state-of-the-art compression for various LLM tensors, enabling a LLaMA-3-70B model to operate within 25GB of memory and projecting up to 4x speedup and 6x energy efficiency for distributed training with minimal accuracy degradation.

Key insights

  • Video codecs' core mechanisms, particularly transform coding and entropy coding, are highly effective at compressing LLM tensors, mitigating outliers and exploiting non-uniform distributions.
  • Hardware video codecs offer a high-throughput, data-independent, and fractional bitrate-capable solution, providing a unified approach to compress diverse LLM tensors without requiring specialized calibration or retraining.
  • Simplified 'tensor codecs' derived from video codec architectures can be integrated into future hardware accelerators, promising substantial speedups and energy efficiency for LLM operations beyond software optimizations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.