Reproduce “Instruction-based Image Manipulation by Watching How Things Move” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingdeng Cao, Xuaner Zhang, Yinqiang Zheng, Zhihao Xia
InstructMove introduces an instruction-based image manipulation model capable of performing complex, non-rigid edits like pose changes and viewpoint alterations. It achieves this by training on a novel 6-million-pair dataset derived from real video frames and employing a spatial conditioning strategy, significantly outperforming prior methods on instruction adherence and content preservation.
Key insights
- Training on real video data, rather than synthetic images, is crucial for developing models that can perform realistic, complex, and non-rigid image transformations while maintaining high fidelity.
- Spatial conditioning, by concatenating reference and noisy inputs across the width, allows the denoising U-Net to process conditions and inputs in parallel and enables cross-attention to attend to any region of the reference features at all layers, improving structural change capabilities and content preservation.
- MLLMs are highly effective for generating precise, context-specific instructions by analyzing visual differences between real image pairs, overcoming limitations of text-only instruction generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.