Reproduce “Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation” on OpenResearch

About this paper

Hanyu Zhou, Gim Hee Lee

Uni4D-LLM from the National University of Singapore presents a unified framework for 4D scene understanding and generation, integrating both autoregressive and diffusion paradigms within a single large multimodal model. This system explicitly incorporates spatiotemporal awareness into its visual representations, achieving competitive performance on benchmarks for 4D perception and content synthesis.

Key insights

• A single Transformer-based LLM architecture can effectively unify both autoregressive understanding and diffusion-based generation paradigms by employing shared weights with task-specific heads and attention masks.
• Integrating multi-level visual features (semantic and noisy appearance) with explicit 4D geometric cues (3D spatial and 1D temporal) via adaptive cross-attention is crucial for a comprehensive, task-aware visual representation.
• The use of dynamic attention masks within the shared LLM backbone and a spatiotemporal alternating attention strategy for 4D generation are vital for controlling information flow and ensuring spatial consistency and temporal continuity.
• Explicit spatiotemporal embedding and adaptive cross-attention fusion significantly improve fine-grained 4D reasoning and content generation quality.

Summary by alphaXiv.