Reproduce “LinearVC: Linear transformations of self-supervised features through the lens of voice conversion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Herman Kamper, Benjamin van Niekerk, Julian Zaïdi, Marc-André Carbonneau

Researchers from Stellenbosch University and Ubisoft La Forge introduced LinearVC, a voice conversion method based on simple linear transformations of self-supervised features. This approach achieves performance comparable to advanced systems while simultaneously revealing that phonetic content and speaker identity are linearly separable within WavLM features, with content residing in a low-dimensional subspace and speaker identity primarily manipulated through rotations and reflections.

Key insights

  • Phonetic content in SSL features is robustly preserved even under various linear transformations, suggesting it resides in a shared, low-dimensional subspace across speakers.
  • Rotations and reflections are the primary linear operations critical for effectively transforming speaker identity within SSL feature space, while simple translation or additional scaling/shearing offer marginal benefits.
  • Speaker and content information can be explicitly disentangled through low-rank factorization, with content requiring a relatively small number of dimensions (e.g., rank 24 for optimal intelligibility).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.