Reproduce “Collaborative Inference for Large Models with Task Offloading and Early Exiting” on OpenResearch

About this paper

Zuan Xie, Yang Xu, Hongli Xu, Yunming Liao, Zhiyuan Yao

DTO-EE provides a collaborative inference framework for large deep learning models across resource-constrained, heterogeneous edge nodes. This approach jointly optimizes task offloading and early exit confidence thresholds in a distributed manner, reducing average response delay by 21-41% and improving inference accuracy by 1-4% compared to baselines on ResNet101 and Bert models.

Key insights

• The coupled relationship between task offloading decisions and early exit confidence thresholds can be jointly optimized in a distributed manner to effectively balance response delay and inference accuracy.
• A distributed optimization approach using local communication and gradient-descent-like updates can adaptively manage complex, dynamic, and heterogeneous edge environments without requiring global state information.
• Introducing a "repulsive factor" in offloading decisions, which considers both a receiver's current load and the future delay fluctuations of its successors, improves load balancing and pipeline efficiency.

Summary by alphaXiv.