Reproduce “DenseMLLM: Standard Multimodal LLMs for Dense Prediction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yi Li, Hongze Shen, Lexiang Tang, Xin Li, Xinpeng Ding, Yinsong Liu, Deqiang Jiang, Xing Sun and 1 more

DenseMLLM enables standard Multimodal Large Language Models to perform high-quality dense prediction tasks, such as semantic segmentation and depth estimation, by using a novel multi-label vision token supervision strategy that eliminates the need for external task-specific decoders. This 4B-parameter model achieves competitive performance across various benchmarks while maintaining general vision-language capabilities.

Key insights

  • Standard MLLM vision tokens inherently contain sufficient information for fine-grained dense prediction tasks, provided they are supervised with a multi-label objective.
  • The Next-Token Prediction paradigm can be effectively extended to vision tokens for multi-label, multi-task scenarios by modeling independent Bernoulli probabilities for each vocabulary ID, rather than a single-label softmax.
  • A relevant negative sampling strategy significantly improves training efficacy by prioritizing harder negative examples, preventing overwhelming gradients from easily distinguishable irrelevant classes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.