Reproduce “Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Luca Della Libera, Cem Subakan, Mirco Ravanelli

DyCAST introduces a speech tokenizer that replaces uniform frame rates with dynamically sized, character-aligned chunks inferred directly from audio at inference time. It reduces token rates to 6.2–17.5 Hz while preserving naturalness and speaker similarity, although aggressive compression increases word-error rates; retrieval-augmented decoding partly recovers lost intelligibility without adding transmitted tokens.

Key insights

  • Character supervision produces speech tokens that are more directly useful for linguistic tasks and enables parallel text-to-speech generation, while predicted boundaries allow transcript-free inference.
  • Compression is adjustable: the system reaches roughly 0.40–1.12 kbps, or 6.2–17.5 Hz, compared with typical 25–50 Hz codecs, but lower rates trade phonetic intelligibility for sequence efficiency.
  • Retrieval is most helpful at the lowest rates, improving reconstruction quality by restoring continuous acoustic detail that quantization removes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.