Reproduce “Towards a Japanese Full-duplex Spoken Dialogue System” on OpenResearch

About this paper

Atsumoto Ohashi, Shinya Iizuka, Jingjing Jiang, Ryuichiro Higashinaka

Nagoya University researchers develop J-Moshi, the first publicly available full-duplex spoken dialogue model for Japanese, by adapting the English Moshi architecture through a two-stage training process that includes pre-training on 60,000 hours of J-CHAT corpus data followed by fine-tuning on 344 hours of high-quality stereo dialogue data, with the extended J-Moshi-ext variant incorporating 602 hours of synthetic dialogue generated via multi-stream TTS achieving significant improvements over the Japanese-trained dGSLM baseline (100-point PPL reduction at τ=0.8) and human evaluation scores of 2.67 for naturalness and 2.30 for meaningfulness, while successfully acquiring Japanese-specific turn-taking characteristics including increased speech overlaps (4.2 seconds per minute vs 1.2 for English Moshi) that align with established linguistic patterns in Japanese conversation, demonstrating effective cross-lingual adaptation of full-duplex dialogue systems despite quality gaps remaining compared to ground-truth audio that highlight areas for future enhancement in the RQ-Transformer component.

Key insights

• Mimi neural audio codec demonstrates language-agnostic capabilities, successfully encoding and resynthesizing Japanese speech without additional training
• Japanese dialogue data exhibits much higher PAD token ratio (88% vs 65% for English) due to denser semantic content in Japanese tokens, suggesting need for language-specific training adjustments
• J-Moshi successfully acquires Japanese-specific conversational characteristics including increased speech overlaps (4.2s/min) that align with established linguistic research on Japanese turn-taking patterns
• Data augmentation using synthetic dialogue generated by multi-stream TTS effectively improves model meaningfulness and language capability

Summary by alphaXiv.