Reproduce “S2M3: Split-and-Share Multi-Modal Models for Distributed Multi-Task Inference on the Edge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
JinYi Yoon, JiHo Lee, Ting He, Nakjung Choi, Bo Ji
S2M3 is a framework designed to deploy advanced multi-modal, multi-task AI models onto resource-constrained edge devices by intelligently splitting models into functional modules and sharing common components across tasks. This approach achieves up to 61.5% reduction in total memory footprint for multi-task deployments and up to 56.9% reduction in inference latency compared to cloud solutions, all while maintaining high accuracy.
Key insights
- Multi-modal models are inherently modular, allowing for functional-level splitting of modality encoders and task-specific heads, which can operate independently for parts of the inference process.
- Different multi-modal tasks often utilize identical underlying encoders (e.g., a common vision encoder), presenting a significant opportunity for module sharing to reduce resource demands.
- In local edge networks, communication latency is often negligible compared to computation time, making distributed processing and parallel execution of modality encoders highly effective for reducing overall latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.