Reproduce “Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, Dmitry Vetrov
COSMOS introduces a text diffusion model that operates within a compressed and smooth latent space, facilitating high-quality text generation. The model achieves performance comparable to state-of-the-art autoregressive models and offers over 2x faster inference compared to token-level diffusion models, especially for longer sequences.
Key insights
- Purely reconstruction-based compressed latent spaces are insufficient for robust generative modeling via diffusion due to brittleness to perturbations.
- Explicitly engineering the latent space for smoothness and robustness through specific training objectives (MSE regularization and various augmentations) is crucial for effective text diffusion.
- Aggressive compression (e.g., 8x for 128 tokens) of text representations is possible without sacrificing generation quality when the latent manifold is properly constructed to be 'diffusable'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.