Reproduce “Efficient Interleaved Speech Modeling through Knowledge Distillation” on OpenResearch

About this paper

Mohammadmahdi Nouriborji, Morteza Rohanian

This research introduces TinyWave, a family of 2-billion-parameter speech generation models, created by applying knowledge distillation from 7-billion-parameter teacher models. The approach achieved an 85–93% reduction in the performance gap to the larger teachers while decreasing parameter count by 3x, making expressive speech modeling feasible on commodity hardware using public datasets.

Key insights

• Layer-aligned distillation effectively transfers complex linguistic and acoustic knowledge from large multimodal transformers to significantly smaller student models (3x parameter reduction) with minimal performance degradation.
• High-quality expressive speech generation, encompassing semantic understanding and prosodic control, can be achieved with compact 2-billion-parameter models using only publicly available datasets.
• Distillation exhibits a trade-off in acoustic consistency, enhancing generalization to diverse acoustic contexts but potentially reducing the preservation of speaker-specific identity.

Summary by alphaXiv.