Reproduce “WorldDiT: A Unified Diffusion Architecture for World and Action Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sen Wang, R. Gnana Praveen, Bidhan Roy, Marcos Villagra
WorldDiT introduces a unified diffusion transformer architecture for robot control that integrates continuous action generation with an auxiliary objective of predicting future normalized RGB patches. This approach achieved a 94.9% mean success rate across LIBERO manipulation benchmarks with a sub-billion-parameter model (399M total), positioning it on the parameter-success Pareto frontier without relying on a large pretrained VLM as an action backbone.
Key insights
- A unified diffusion transformer can effectively couple continuous action generation with future visual prediction, using world modeling as a supervisory signal during training.
- Strong robot control performance, comparable to or exceeding methods employing large pretrained VLMs, can be achieved with a substantially smaller parameter count (sub-billion).
- The integration of an auxiliary future visual prediction objective provides valuable guidance to the action generation process, even if the visual prediction component is not used at inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.