Reproduce “UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yaqi Zhao, Wang Lin, Zijian Zhang, Miles Yang, Jingyuan Chen, Wentao Zhang, Zhao Zhong, Liefeng Bo

The UniCom framework unifies multimodal understanding and generation by introducing compressed continuous semantic representations, achieving high-fidelity image reconstruction and competitive performance in text-to-image generation and image editing tasks. This method effectively bridges the gap between preserving fine-grained details and ensuring generative tractability by employing an attention-based channel-wise compression strategy.

Key insights

  • Channel-wise compression of continuous visual features is more effective than sequence length reduction for preserving fine-grained details and accelerating generative model convergence.
  • An attention-based projector in the Semantic Compressor is crucial for maintaining structured semantic information from visual inputs, outperforming simpler MLP designs.
  • Pre-training with visual-language alignment from VLMs significantly bootstraps and enhances the generative process, leading to faster convergence and higher performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.