Reproduce “Audio-Visual World Models: Learning Physically Grounded Multisensory Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiahua Wang, Leqi Zheng, Jialong Wu, Yaoxin Mao, Shijie Cheng
AVWM enables an agent to predict future views and binaural audio under candidate movements. AV-CDiT jointly predicts synchronized streams conditioned on motion and time, using modality experts and staged training. On simulated indoor scenes with stationary sound sources, it improves consistency proxies over an adapted audiovisual baseline. In one tested navigation configuration, planning takes 317.19 actions versus 332.817 for the baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.