Reproduce “From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pan Wang, Siwei Song, Hui Ji, Siqi Cao, Heng Yu, Zhijian Liu, Huanrui Yang, Yingyan Celine Lin and 7 more

Efficient multimodal learning is organized as a full-stack co-design problem spanning model architecture, inference algorithms, and deployment systems. A survey of more than 300 studies develops the Model–Algorithm–System taxonomy, showing how token reduction, quantization, caching, scheduling, and hardware-aware execution jointly affect accuracy, latency, memory, energy, communication, and privacy.

Key insights

  • Efficiency gains depend on interactions between layers: token pruning works better when representations expose redundancy, quantization benefits from reduced cross-modal distribution mismatch, and algorithmic savings require suitable kernels, memory systems, and scheduling to produce practical acceleration.
  • The field has progressed from reducing model parameters, to accelerating inference, to orchestrating resources dynamically according to semantic difficulty, uncertainty, workload, and available hardware.
  • The most informative evaluation should report end-to-end latency, peak memory, energy per inference, communication and synchronization overhead, and privacy alongside task accuracy rather than relying on FLOPs or compression ratios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.