Reproduce “Rethink Arbitrary Style Transfer with Transformer and Contrastive Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhanjie Zhang, Jiakai Sun, Guangyuan Li, Lei Zhao, Quanwei Zhang, Zehua Lan, Haolin Yin, Wei Xing and 2 more

A new arbitrary style transfer method integrates a Perception Encoder, Style Consistency Instance Normalization, and Instance-based Contrastive Learning within a unified network to produce high-quality stylized images. The approach yielded a deception score of 0.573, indicating human subjects mistake its outputs for real artworks more often than outputs from other state-of-the-art methods.

Key insights

  • VGG networks are suboptimal for style feature extraction due to their bias towards classification features, necessitating a dedicated encoder to effectively capture artistic styles.
  • Explicitly modeling 'stylization-to-stylization' relationships via contrastive learning significantly improves the consistency, quality, and artifact suppression of generated images.
  • A transformer-based instance normalization, using learnable parameters derived from global style, achieves a more refined and adaptive feature alignment than traditional statistical methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.