Reproduce “Benchmarking of DL Libraries and Models on Mobile Devices” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qiyang Zhang, Xiang Li, Xiangying Che, Xiao Ma, Ao Zhou, Mengwei Xu, Shangguang Wang, Yun Ma and 1 more

A comprehensive benchmark, MDLBench, was developed to evaluate deep learning library performance on diverse mobile devices. The study reveals significant performance fragmentation, with up to 54.3x differences between libraries for specific models, and identifies that cold-start inference can be 1.3x to 45.0x slower due to memory preparation overheads.

Key insights

  • A significant performance fragmentation exists in the mobile DL ecosystem, with no single library consistently outperforming others across all models and devices.
  • INT8 quantization provides limited speedup (0.8x-3.0x) and can occasionally lead to slower inference compared to FP32, challenging assumptions about its universal benefits.
  • Cold-start inference introduces substantial overhead (1.3x-45.0x slower than warm inference), primarily due to single-threaded memory preparation.
  • Different DL libraries demonstrate specialized strengths at the operator level, suggesting potential gains from integrating best-performing operators across libraries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.