Reproduce “StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wen Li, Muyuan Fang, Cheng Zou, Biao Gong, Ruobing Zheng, Meng Wang, Jingdong Chen, Ming Yang
Researchers at Ant Group developed StyleTokenizer, a method enabling precise visual style control in diffusion models from a single reference image while maintaining strong adherence to text prompts. The approach integrates style information directly into the text embedding space, achieving clean separation of style and content control, and it outperforms prior methods in quantitative metrics and user studies for style similarity, text adherence, and aesthetic quality.
Key insights
- Aligning style representations directly within the text embedding space allows diffusion models to naturally integrate style and content control without interference.
- A specialized Style Encoder, trained contrastively on a meticulously curated dataset (Style30K) of diverse styles with content-only annotations, is crucial for robustly capturing style nuances while disentangling them from content.
- Mapping style features to style tokens that concatenate with text tokens enables the existing cross-attention mechanisms of diffusion models to process both harmoniously.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.