Reproduce “Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kamran Razavi, Saeid Ghafouri, Max Mühlhäuser, Pooyan Jamshidi, Lin Wang

Sponge is a deep learning inference serving system developed by researchers from TU Darmstadt, QMUL, University of South Carolina, and Paderborn University. It addresses the challenge of guaranteeing dynamic, per-request Service Level Objectives (SLOs) in mobile and IoT environments by employing in-place vertical scaling, dynamic batching, and request reordering. The system achieved less than 0.3% SLO violations while reducing CPU allocation by over 20% compared to static over-provisioning and significantly outperforming horizontal autoscalers under fluctuating network conditions.

Key insights

  • Integrating end-to-end latency, including highly variable network transmission time, into the server-side SLO calculation is crucial for reliable deep learning inference in mobile/IoT settings.
  • In-place vertical scaling effectively eliminates cold-start issues, enabling rapid resource adaptation that traditional horizontal autoscaling cannot achieve for dynamic SLOs.
  • A combined approach of dynamic resource allocation, intelligent request scheduling, and optimized batching can significantly improve both SLO adherence and resource efficiency.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.