Reproduce “MNN: A Universal and Efficient Inference Engine” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui and 4 more
MNN is a deep learning inference engine designed for mobile applications that employs a semi-automated search architecture and optimized kernels to achieve high universality and efficiency. It consistently outperforms other mobile inference solutions by 20% to 40% across diverse devices and network architectures, and is deployed at scale within Alibaba's mobile services.
Key insights
- A semi-automated search architecture effectively balances the efficiency of hand-optimized engines with the universality and scalability of automated compilers by integrating runtime optimization and comprehensive kernel tuning.
- Dynamic runtime cost evaluation and preparation-execution decoupling can significantly reduce inference overhead by selecting optimal algorithms and hardware backends while pre-allocating memory for heterogeneous devices.
- Universal operator generators (e.g., for Winograd convolution) and adaptive algorithms (e.g., dynamic Strassen for matrix multiplication) enhance scalability across diverse models and prevent performance bottlenecks from unoptimized 'special operators'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.