Reproduce “JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Siheng Wan, Zhengtao Yao, Zhengdao Li, Junhao Dong, Yanshu Li, Yikai Li, Linshan Li, Haoyan Xu and 4 more

JEPA-T adds language conditioning to visual-token generation through two stages: text guides prediction of missing visual features, then cross-attention and direct text-feature fusion refine them before latent flow matching. On ImageNet-1K at 256×256, the full model reports FID 1.42, compared with 1.75 without cross-attention. This ablation supports cross-attention’s role in that setup, but does not establish robust free-text generation.

Key insights

  • The multi-stage text fusion strategy, which includes late architectural cross-attention and objective-level text injection, effectively balances the preservation of a general visual backbone with strong semantic conditioning.
  • Joint-Embedding Predictive Architectures can be successfully adapted for complex multimodal generative tasks, demonstrating their versatility beyond pure self-supervised vision.
  • Combining masked prediction with conditional flow matching provides a robust and scalable training methodology for high-quality, semantically consistent text-to-image generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.