Reproduce “Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Davide Berghi, Philip J. B. Jackson
Researchers at the University of Surrey integrated language-aligned embeddings from CLAP and OWL-ViT into a stereo Sound Event Localization and Detection (SELD) system for videos using Cross-Modal Conformers. Their method achieved an F1 score of 48.0% and a Relative Distance Error (RDE) of 29.3%, securing second rank in the DCASE 2025 Task 3 challenge.
Key insights
- Pre-trained, language-aligned models (CLAP, OWL-ViT) can effectively enrich SELD systems with semantic reasoning capabilities, indirectly improving spatial and temporal understanding in multimodal contexts.
- Generating and utilizing large synthetic audio and audio-visual datasets for pre-training significantly enhances model performance and generalization, addressing limitations of real-world data scarcity.
- The proposed Cross-Modal Conformer architecture provides a robust method for fusing disparate feature embeddings from standard SELD encoders with semantic representations, crucial for complex multimodal tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.