Reproduce “DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kevin Wilkinghoff, Zheng-Hua Tan
Researchers from Aalborg University and Pioneer Centre for AI developed DSpAST, a spatial audio encoder that learns disentangled representations for sound event detection, distance prediction, and direction-of-arrival estimation. This architecture improves performance across all three tasks compared to SpatialAST, achieving a 54.53% mAP for SED and 20.28% ER_20° for DoAE, while adding only 0.2% more parameters.
Key insights
- Learning disentangled representations for multiple spatial audio tasks within a single encoder can significantly improve individual task performance.
- Feature attention modules enable selective utilization of spatial audio features; for instance, GCC-PHAT is crucial for distance and direction but not for sound event detection.
- A staged training curriculum, progressively focusing on different loss components, effectively optimizes the disentangled encoder.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.