Reproduce “HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guozhen Zhang, Xuerui Qiu, Yutao Cui, Tianhui Song, Changlin Li, Junzhe Li, Tao Huang, Xiao Zhang and 6 more

The HYDRA-X framework presents a unified multimodal model leveraging HYDRA-XTOK, a holistic visual tokenizer that natively integrates images and videos within a single Vision Transformer. This integration enables strong performance across visual understanding, generation, and image editing tasks, outperforming comparable baselines on reconstruction, generation (e.g., VBench total score +2.15 over Show-o2-1.5B), and significantly improving image editing consistency.

Key insights

  • For video processing within an image-pretrained Vision Transformer, restricted frame-level causal tubelet attention and hierarchical temporal patchify significantly improve reconstruction fidelity over broader attention or single-step compression.
  • A lightweight Decompressor module, used with dual image-video teacher distillation, effectively imbues compact video latents with explicit spatiotemporal semantic structure, which is indispensable for strong video understanding.
  • Repurposing video's temporal-causal attention mechanism for tokenizer-stage source-target interaction in image editing enhances structural consistency and identity preservation by enabling early, latent-level interaction.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.