Reproduce “PolyMaX: General Dense Prediction with Mask Transformer” on OpenResearch

About this paper

Xuan Yang, Liangzhe Yuan, Kimberly Wilber, Astuti Sharma, Xiuye Gu, Siyuan Qiao, Stephanie Debats, Huisheng Wang and 3 more

PolyMaX generalizes the mask transformer's cluster-prediction paradigm to diverse dense prediction tasks, including continuous outputs like depth and surface normals, by discretizing output spaces. This unified framework achieves state-of-the-art results on the NYUD-v2 dataset for semantic segmentation, depth estimation, and surface normal prediction.

Key insights

• The cluster-prediction paradigm of mask transformers, combined with adaptive output space discretization, can effectively generalize to continuous and multi-dimensional dense prediction tasks.
• A single, unified mask transformer architecture can achieve state-of-the-art performance across diverse dense prediction tasks (semantic segmentation, depth, surface normals), often surpassing task-specific models.
• PolyMaX-like architectures demonstrate superior scalability with increased pretraining data compared to conventional per-pixel regression methods, highlighting their advantage for large-scale learning.

Summary by alphaXiv.