Reproduce “Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yabo Zhang, Kunchang Li, Dewei Zhou, Xinyu Huang, Xun Wang
Inset, a unified visual generation model from ByteDance Seed, directly integrates images into textual instructions as native vocabulary, supported by a scalable data engine. It achieves superior image and text consistency on complex multi-image tasks, outperforming open-source baselines by notable margins on the new InterleaveBench.
Key insights
- Directly embedding visual tokens within instructions, rather than using indirect referencing, significantly improves object binding and attribute consistency by simplifying long-range dependencies.
- Training with video-based interleaved data is crucial for enabling the model to dynamically alter object attributes and synthesize natural, interactive dynamics based on textual instructions.
- Exclusively relying on semantic ViT embeddings, without VAE latent features, prevents 'image-pasting' biases and enhances overall consistency by encouraging deeper semantic reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.