Reproduce “NVIDIA Nemotron Nano V2 VL” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

NVIDIA, Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov and 115 more

NVIDIA introduces Nemotron Nano V2 VL, an efficient 12B vision-language model integrating a hybrid Mamba-Transformer architecture and innovative training to achieve strong performance in real-world document understanding, long video comprehension, and complex reasoning tasks. This model extends effective context length up to 300K tokens and enhances inference throughput by 35% for documents and 2x for videos, with open-source model weights and datasets provided.

Key insights

  • Hybrid Mamba-Transformer architectures can achieve high inference throughput for long-context scenarios in VLMs while maintaining strong performance, offering an alternative to purely Transformer-based models.
  • A multi-stage training strategy with specific phases for context extension and text reasoning recovery is crucial for building robust VLMs that excel across diverse multimodal and text-only tasks without catastrophic forgetting.
  • Token reduction techniques, such as Efficient Video Sampling, effectively boost video understanding inference throughput significantly (e.g., 2x or more) with minimal impact on accuracy, making video processing more practical.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.