Reproduce “Real-Time Text Transmission via LLM-Based Entropy Coding over Fixed-Rate Channels” on OpenResearch

About this paper

Vishnu Teja Kunde, Jean-Francois Chamberland, Krishna R. Narayanan, Jamison Ebert

This research investigates the compression-delay tradeoff in real-time text transmission utilizing Large Language Models (LLMs) as sequential predictors. It demonstrates that higher quality LLMs, such as Llama 3.2, reduce the bits per character by an average of 38% compared to GPT-2, which fundamentally alters system dynamics and makes simpler, low-delay coders like Huffman optimal in many real-time scenarios.

Key insights

• The quality of the LLM predictor profoundly impacts system performance; higher-quality models significantly reduce the effective source rate, which in turn shifts the optimal choice of source coder.
• In over-provisioned channel scenarios, Huffman coding, when combined with a high-quality LLM predictor, emerges as a pragmatic choice due to its zero algorithmic delay, simple implementation, and sufficiently efficient compression.
• Modern tokenization schemes (like BPE and SentencePiece) inherently group characters into larger units, which substantially amortizes Huffman coding's per-symbol overhead, making it highly competitive against more complex block-based entropy coders.

Summary by alphaXiv.