Reproduce “PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zheng Zhang, Yeyao Ma, Enming Zhang, Xiang Bai
PSALM integrates a Large Multi-modal Model with an external mask decoder to perform diverse pixel-level segmentation tasks within a single unified framework. The model achieves competitive performance on referring and panoptic segmentation while demonstrating strong zero-shot generalization across open-vocabulary and video object segmentation benchmarks.
Key insights
- Large Multi-modal Models can be extended to precise pixel-level tasks by decoupling their textual reasoning from mask generation via an external decoder.
- A unified framework with a flexible input schema allows a single model to handle a wide range of diverse segmentation tasks, eliminating the need for specialized models.
- Joint training across diverse segmentation tasks and vision-language data significantly enhances a model's generalization, leading to strong zero-shot performance on unseen tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.