Reproduce “SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman
A new benchmark, SAVVY-Bench, is introduced for evaluating 3D spatial reasoning in dynamic audio-visual environments, alongside SAVVY, a training-free pipeline that enhances existing AV-LLMs by integrating spatial audio and visual information within a global coordinate framework, improving Gemini-2.5-pro's overall accuracy by 7.1%.
Key insights
- Integrating spatial audio and visual information within a global coordinate framework is essential for human-like 3D spatial understanding in dynamic scenes.
- Current AV-LLMs face significant limitations in temporal grounding and object referral, hindering their 3D spatial reasoning abilities.
- A training-free, modular pipeline can substantially enhance the spatial reasoning capabilities of existing foundation models by explicitly modeling spatial relationships.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.