Reproduce “Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham, Arman Luthra

Fusion Embedding adds audio retrieval to a frozen vision–language model through a trained connector from a frozen audio encoder; gated adapters can add capacity for audio inputs. On held-out VGGSound-696, generation 1 reaches audio-to-image R@10 0.418, versus 0.014 chance, without audio–image training pairs. This shows emergent retrieval on one benchmark, not universal audio–visual alignment.

Key insights

  • A frozen-base fusion paradigm can effectively extend multimodal capabilities by aligning a new modality (audio) to an existing pivot (text), inducing emergent cross-modal retrieval (e.g., audio-image) without direct paired training.
  • Training and evaluating using the base LLM's native input protocol (chat template) is critical for optimal performance, leading to substantial retrieval accuracy improvements compared to using bare input sequences.
  • In-layer capacity additions via modality-gated adapters can improve performance for the new modality without compromising the computational invariance or parameter integrity of the original frozen backbone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.