Reproduce “Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini and 1 more
NCM generates text by encoding it into hierarchical discrete representations and predicting them coarse to fine, with codes at each level generated in parallel. It reports LM1B generative perplexity of 93.05 at 12 function evaluations, compared with DUO’s 97.30 at 128, alongside similar measured diversity and a reported 21× wall-clock speedup. These results are benchmark-specific, and larger-scale behavior remains open.
Key insights
- The method reduces both the number of sequential generation stages and the work per stage by processing short coarse representations before finer ones.
- The tokenizer reconstructs text with high exact-token accuracy on the reported datasets: 99.86–99.99% for three language-modeling datasets, 99.79% on XSum, and 98.79% on WMT14.
- Low-cost generation can be paired with candidate selection: minimum Bayes risk selection improved summarization and translation scores when more inference evaluations were allocated.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.