Reproduce “Fine-Grained and Interpretable Neural Speech Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Max Morrison, Cameron Churchwell, Nathan Pruyne, Bryan Pardo
Researchers at Northwestern University developed a system for fine-grained and interpretable neural speech editing by introducing a novel disentangled speech representation and training an off-the-shelf neural vocoder with specialized data augmentation. This approach allows precise control over attributes like pitch, loudness, pronunciation, and spectral balance, achieving perceptual quality comparable to Mel spectrograms.
Key insights
- A carefully engineered, disentangled speech representation can achieve high-quality speech reconstruction while enabling fine-grained, independent control over multiple speech attributes.
- Targeted data augmentation during vocoder training effectively expands the editing range and allows for the disentanglement of perceptually linked attributes, such as spectral balance from fundamental frequency and perceived volume from its timbral characteristics.
- Optimizations like Viterbi decoding for pitch estimation and variable-width pitch quantization significantly improve the stability and accuracy of prosodic control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.