Reproduce “On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation” on OpenResearch

About this paper

Changhao Cheng, Wei Wang, Wangyou Zhang, Dongya Jia, Jian Wu, Zhuo Chen, Yanmin Qian

Researchers from Shanghai Jiao Tong University and ByteDance Seed developed Variational Autoencoder (VAE) distillation loss functions and an adaptive weighting strategy to create continuous speech representations capable of balanced performance across speech reconstruction, understanding, and generation tasks. Their Joint-marginal Aligned Semantic VAE (JMAS-VAE) with adaptive weighting achieved the highest overall performance by effectively integrating both frame-level semantic and sequence-level acoustic information.

Key insights

• Dimension-wise alignment (DAS-VAE) and particularly joint-marginal distribution alignment (JMAS-VAE) are more effective than simple time-axis alignment for creating VAE representations that also excel in speech understanding, not just reconstruction and generation.
• Adaptive weighting of distillation losses, which dynamically adjusts weights based on gradient norms, is crucial for achieving a balanced performance across reconstruction, understanding, and generation, particularly for the JMAS-VAE.
• The two components of joint-marginal alignment—frame-level marginal cosine similarity (L_mcos) and sequence-level marginal distance sequence similarity (L_mdss)—play distinct roles, with L_mcos emphasizing semantic information and L_mdss preserving acoustic information, requiring careful balancing for optimal unified performance.

Summary by alphaXiv.