Reproduce “Splicing ViT Features for Semantic Appearance Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, Tali Dekel
Researchers from the Weizmann Institute of Science developed "Splice," a method for semantic appearance transfer that leverages features from pre-trained Vision Transformers. Splice operates on just a single image pair, effectively transferring appearance between semantically related objects while preserving structural integrity and outperforming previous techniques.
Key insights
- Features from pre-trained DINO-ViT models contain rich semantic information, with the [CLS] token encoding high-level appearance and the self-similarity of keys capturing structural and semantic relationships independent of specific appearances.
- By separating appearance and structure in the ViT feature space, effective semantic appearance transfer can be achieved using a generator trained on a single image pair.
- Extensive data augmentation on a single image pair enables robust generalization during training, making the method practical without large datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.