Reproduce “PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman, Daan de Geus
The Plain Mask Transformer (PMT) introduces an efficient architecture for image and video segmentation that leverages frozen Vision Foundation Model (VFM) encoders. It achieves comparable or superior accuracy to existing state-of-the-art methods while offering significantly faster inference, with up to an 8x speedup in video segmentation, thereby enabling multi-task deployment from a single, shared VFM.
Key insights
- The incompatibility of existing encoder-only models with frozen encoders stems from the inability of pre-trained attention layers to effectively process newly injected query tokens without finetuning.
- A dedicated, lightweight decoder (PMD) can effectively mimic the query-patch attention mechanisms of finetuned encoder-only models, restoring performance with a frozen encoder.
- Accessing multi-scale features via lateral connections and using relative positional embeddings (RoPE) are crucial for compensating for the frozen encoder's lack of task-specific adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.