Reproduce “Disentangled Acoustic Fields For Multimodal Physical Scene Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jie Yin, Andrew Luo, Yilun Du, Anoop Cherian, Tim K. Marks, Jonathan Le Roux, Chuang Gan
DAF helps a robot find objects by comparing sounds predicted for candidate locations with an impact recording. Its encoder estimates object type, material, position, and latent factors, while a generator predicts power spectral density; reconstruction error ranks locations for visual search. In TDW simulation, full DAF achieved 0.57 success versus 0.41 for modular planning, in kitchens and study rooms.
Key insights
- Modeling acoustic formation with disentangled factors (position, material, type, latent code) enables superior generalization across diverse acoustic environments.
- Power Spectral Density (PSD) provides a more robust and learnable acoustic representation for irregular impact sounds compared to Short-Time Fourier Transforms (STFTs).
- An analysis-by-synthesis approach that generates spatial uncertainty maps allows for more intelligent and robust embodied navigation strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.