Reproduce “6DoF SELD: Sound Event Localization and Detection Using Microphones and Motion Tracking Sensors on self-motioning human” on OpenResearch

About this paper

Masahiro Yasuda, Shoichiro Saito, Akira Nakayama, Noboru Harada

Researchers at NTT Corporation developed a novel 6DoF SELD framework that localizes and detects sound events for a self-motioning human using wearable microphones and motion tracking sensors. The system leverages a new 20.1-hour multimodal dataset and integrates motion data to excite acoustic features, achieving an F1-score of 54.1% and an error rate of 0.51 in dynamic environments.

Key insights

• Training SELD systems with data that explicitly includes self-motion conditions is crucial for achieving robust performance in dynamic environments, as static-trained systems suffer significant degradation.
• Integrating velocity and angular velocity data from head motion tracking sensors into SELD systems consistently improves performance for self-motioning users compared to audio-only approaches.
• Modulating acoustic features with motion sensor signals through an excitation mechanism (MMTM) is more effective than simple feature concatenation, enabling the system to better utilize dynamic acoustic cues.

Summary by alphaXiv.