Reproduce “Adaptive Compression-Aware Split Learning and Inference for Enhanced Network Efficiency” on OpenResearch

About this paper

Akrit Mudvari, Antero Vainio, Iason Ofeidis, Sasu Tarkoma, Leandros Tassiulas

Researchers from Yale University and the University of Helsinki developed an adaptive compression-aware split learning framework that dynamically adjusts compression levels for deep learning models at the wireless edge. This framework, using `deprune` and `prune` methods, significantly reduces network usage during distributed training and accelerates the generation of models for efficient inference.

Key insights

• A novel loss function allows parallel optimization of model accuracy and communication budget through a single learnable filter parameter `f`, enabling fine-grained control over compression.
• Dynamic adjustment of the communication budget during distributed training, coupled with temporary learning rate boosts, enables efficient distributed training with reduced network load and improved end-to-end latency.
• Transfer learning from a higher-fidelity, fully trained model allows for rapid generation of lower-fidelity (higher compression) models suitable for diverse inference scenarios, avoiding costly retraining from scratch.

Summary by alphaXiv.