Reproduce “UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles and 5 more
UniControl from Salesforce AI Research, Northeastern University, and Stanford University introduces a single diffusion model to manage diverse condition-to-image tasks, achieving precise pixel-level control while significantly reducing model parameters. It demonstrates superior image quality and robust zero-shot generalization across various visual conditions and novel tasks.
Key insights
- It is feasible to unify diverse visual control tasks into a single diffusion model, leading to substantial parameter efficiency compared to an ensemble of specialized models.
- Explicitly guiding the model with natural language task instructions via a Task-Aware HyperNet enables effective meta-knowledge learning and strong zero-shot generalization capabilities.
- A Mixture-of-Experts style adapter efficiently handles the distinct low-level feature requirements of various visual conditions while maintaining overall model compactness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.