Reproduce “Variable-rate discrete representation learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sander Dieleman, Charlie Nash, Jesse Engel, Karen Simonyan
A framework learns variable-rate discrete representations for perceptual signals like speech, automatically adapting computational capacity to semantic information density rather than fixed rates. This approach uses Slow Autoencoders for representation learning and Run-length Transformers for generative modeling, demonstrating strong correlation with phoneme boundaries and efficient speech generation.
Key insights
- Meaningful changes in perceptual signals occur at irregular intervals, enabling computational cost to scale with semantic density rather than raw signal duration.
- Schmitt Trigger Quantization (STQ) is essential for generating stable, discrete representations by preventing rapid oscillations and reducing spurious events, thus improving compressibility.
- Group-sparse (GS) slowness penalties effectively encourage synchronized, burst-like changes across multiple channels, creating highly structured and compressible representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.