Reproduce “Cosmos 3: Omnimodal World Models for Physical AI” on OpenResearch

About this paper

NVIDIA, Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini and 287 more

Cosmos 3 is a configurable model family that combines autoregressive reasoning and diffusion generation in one multimodal architecture, sharing attention while keeping their parameter paths separate. By changing conditioned and generated modalities, it supports scene understanding, future prediction, and action prediction. On RoboLab-120, a DROID-post-trained Nano policy reaches 39.7% success, versus 30.2% for PT-init, not one universal checkpoint.

Key insights

• Unifying understanding and generation within a single omnimodal architecture allows for shared representations and joint multi-task supervision, improving scalability and efficiency for Physical AI.
• The dual-tower Mixture-of-Transformers effectively compartmentalizes reasoning and generation pathways while leveraging a common architecture, enabling specialized processing within a unified model.
• A progressive, multi-stage data curriculum, incrementally introducing modalities and specializing domains through fine-tuning, is crucial for developing high-performing and adaptable omnimodal world models.

Summary by alphaXiv.