Reproduce “ELF: Embedded Language Flows” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Keya Hu, Linlu Qiu, Yiyang Lu, Hanhong Zhao, Tianhong Li, Yoon Kim, Jacob Andreas, Kaiming He
The paper presents Embedded Language Flows, a system that refines noisy passages in continuous embedding space and converts them to discrete words only at the end, using a shared denoiser-decoder. On OpenWebText, its 105-million-parameter ELF-B reaches generative perplexity 24.08 in 32 sampling steps, with entropy 5.15. This is not a measured speedup or one-step result.
Key insights
- Continuous Diffusion Language Models can achieve competitive, even superior, performance to discrete DLMs with minimal adaptation to the discrete nature of language.
- The `x`-prediction strategy, combined with a bottleneck architecture and a logit-normal time schedule, provides stable training and robust performance for language embeddings.
- Techniques like Classifier-Free Guidance (CFG) and SDE-inspired samplers, common in continuous domains, effectively improve text quality and efficiency in continuous language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.