Reproduce “UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes” on OpenResearch

About this paper

Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, Neil Houlsby

Google Research's UViM presents a unified modeling framework for diverse computer vision tasks by learning a compact, discrete "guiding code" that an autoregressive language model predicts from an image, which then informs a feed-forward Vision Transformer to generate high-dimensional outputs. This approach achieves competitive performance, for instance, a Panoptic Quality of 45.8 on COCO Panoptic Segmentation and an RMSE of 0.467 on NYU Depth v2 for monocular depth estimation, utilizing Vision Transformer components throughout.

Key insights

• High-dimensional structured visual outputs can be accurately reconstructed by a feed-forward model if provided with a sufficiently informative, yet compact, learned discrete "guiding code."
• Autoregressive language models, adapted from NLP, can effectively learn to predict these compact discrete guiding codes from visual inputs, overcoming the high-dimensionality challenge of direct pixel modeling.
• Regularization techniques like "code dropout" during the first training stage are crucial to ensure the learned guiding codes are robust and learnable by the subsequent language model.

Summary by alphaXiv.