Reproduce “Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution” on OpenResearch

About this paper

Qihao Liu, Xi Yin, Alan Yuille, Andrew Brown, Mannat Singh

CrossFlow introduces a noise-free framework for cross-modal media generation by directly evolving one modality's latent representation into another's, bypassing the need for Gaussian noise or explicit cross-attention. It achieves competitive performance in text-to-image generation, outperforming standard flow matching baselines, and generalizes effectively across tasks like image captioning and depth estimation.

Key insights

• Generative models can effectively learn direct mappings between correlated cross-modal latent spaces, potentially eliminating the need for Gaussian noise and explicit conditioning mechanisms like cross-attention.
• A regularized, semantically rich source latent space, often achieved through a Variational Encoder with a contrastive loss, is crucial for high-quality direct cross-modal evolution.
• A unified, simpler architecture (vanilla Transformer) can generalize across diverse cross-modal and intra-modal tasks without task-specific modifications.

Summary by alphaXiv.