Reproduce “Data-Efficient Multimodal Fusion on a Single GPU” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C. Cresswell, Guangwei Yu, Gabriel Loaiza-Ganem, Maksims Volkovs

This paper presents a framework for data-efficient multimodal fusion that utilizes frozen pre-trained unimodal encoders and lightweight fusion adapters, enabling competitive performance on tasks like image-text and audio-text retrieval while significantly reducing computational resources and paired data requirements. The approach incorporates FuseMix, a novel latent-space data augmentation strategy, and demonstrates the framework's ability to facilitate cross-modal applications like audio-to-image generation.

Key insights

  • Pre-trained unimodal encoders contain rich semantic representations that can be effectively leveraged as a bootstrap for multimodal fusion, significantly reducing training data and computation requirements.
  • Data augmentation operating in the latent space (FuseMix) can effectively enhance data efficiency for multimodal tasks by generating semantically consistent interpolated samples.
  • Prioritizing dataset quality and diversity, especially in data-scarce scenarios, yields more substantial performance improvements than merely increasing raw data quantity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.