Reproduce “Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech” on OpenResearch

About this paper

Hongfei Du, Jiacheng Shi, Sidi Lu, Gang Zhou, Ye Gao

Researchers at William & Mary developed a framework using Sparse Autoencoders (SAEs) applied to the semantic backbone of text-to-speech (TTS) models, enabling interpretable, bidirectional emotion control by manipulating sparse latent features. This method achieved high emotion similarity, maintained speech naturalness with a human evaluation EMOS of 3.22 and NMOS of 3.49, and preserved linguistic fidelity more effectively than global steering methods.

Key insights

• Emotional modulation is sparsely organized within the latent space of semantic representations, concentrated in a small subset of latent features rather than being broadly distributed.
• Interventions on individual, selectivity-ranked latent features produce measurable and interpretable acoustic effects, such as localized mid- to high-frequency spectral amplification, increased mean F0, and RMS energy, as well as modulation of spectral brightness.
• Distinct sparse latent features are primarily responsible for different emotions at high activation levels, with no observed overlap among the top-6 selected features across various emotions.

Summary by alphaXiv.