Reproduce “Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michael Cunningham
This research introduces a privacy-aware split inference system for large language models over wide-area networks, enabling regulated industries to leverage powerful cloud LLMs without exposing raw data. The system integrates lookahead decoding to effectively amortize network latency, achieving throughput of 13-17 tokens/second while maintaining output quality and empirically quantifying the privacy-performance tradeoff.
Key insights
- Repurposing speculative decoding techniques like lookahead decoding is effective for mitigating WAN latency in split inference for LLMs, pushing throughput towards interactive speeds.
- An architectural split that keeps raw token IDs and embedding/unembedding layers on a local device provides a practical level of privacy by preventing direct exposure of sensitive input data to the cloud.
- There is an empirical privacy-performance tradeoff: increasing the number of local layers significantly reduces the success rate of inversion attacks on intermediate activations with minimal impact on overall throughput.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.