Reproduce “Sequential Modeling Enables Scalable Learning for Large Vision Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, Alexei A Efros

A unified representation called "visual sentences" enables large vision models to learn directly from diverse pixel data through sequential modeling, using a next-token prediction objective. This approach facilitates learning from varied visual inputs like images, videos, and annotations without language supervision or task-specific training.

Key insights

  • A unified "visual sentence" representation allows large vision models to learn general visual concepts from diverse data using simple sequential prediction.
  • The sequential modeling approach exhibits consistent scaling behavior with increasing model size and data, achieving lower perplexity across various vision tasks.
  • Visual prompting enables the model to perform a wide range of vision tasks at inference time without task-specific fine-tuning, demonstrating strong out-of-distribution generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.