Reproduce “TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms” on OpenResearch

About this paper

Yueyuan Sui, Minghui Zhao, Junxi Xia, Xiaofan Jiang, Stephen Xia

Researchers from Northwestern and Columbia Universities developed TRAMBA, a hybrid deep learning architecture combining Transformer and Mamba components, for speech super-resolution and enhancement. Optimized for mobile and wearable platforms, it achieves high perceptual speech quality with a significantly reduced memory footprint (19.7 MB) and faster inference (2.33 ms for 512 ms window) compared to existing methods, while effectively enhancing bone conduction and accelerometer speech after minimal user-specific fine-tuning.

Key insights

• A hybrid deep learning architecture that judiciously combines elements from U-Net, Transformers (SAFiLM), and State Space Models (Mamba) can achieve high speech enhancement performance while maintaining low computational requirements.
• Pre-training on abundant over-the-air audio data, followed by targeted user-specific fine-tuning, is an effective strategy to overcome the data scarcity challenge for vibration-based speech modalities.
• Enhanced vibration-based speech can offer superior intelligibility and perceived quality in extremely noisy environments compared to traditional OTA microphone systems, even with noise reduction.
• Reducing the sampling rate of vibration sensors does not significantly compromise TRAMBA's enhancement quality, enabling substantial power savings and extended battery life for wearable devices.

Summary by alphaXiv.