Reproduce “SCALAR: Scale-wise Controllable Visual Autoregressive Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ryan Xu, Dongyang Jin, Yancheng Bai, Rui Lan, Xu Duan, Lei Sun, Xiangxiang Chu

This research from Amap, Alibaba Group introduces SCALAR and SCALAR-Uni, a new framework for controllable image synthesis that leverages Visual Autoregressive (VAR) models. The method achieves superior image generation quality and competitive control consistency by integrating fine-grained, scale-wise control signals from robust vision foundation models into the VAR's hierarchical decoding process, also supporting unified multi-conditional control and exhibiting strong zero-shot generalization.

Key insights

  • Scale- and layer-specific projection blocks are critical for optimally tailoring control signals to the hierarchical structure of VAR models, improving performance over shared parameters.
  • Injecting control features into all layers of the VAR decoder and fine-tuning all VAR parameters significantly enhances both image generation quality and conditional consistency.
  • Leveraging robust, pretrained vision foundation models like DINOv2 as control encoders, especially by using multi-layer features, provides superior semantic understanding and boosts overall performance.
  • An explicit alignment loss (L_align) effectively maps diverse control modality features into a shared latent space, which is crucial for robust unified multi-conditional control within a single model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.