Reproduce “Context-level Language Modeling by Learning Predictive Context Embeddings” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Beiya Dai, Yuliang Liu, Daozheng Xue, Yunchong Song, Qipeng Guo, Kai Chen, Xinbing Wang, Bowen Zhou and 1 more

ContextLM forecasts a latent representation of the next text chunk and uses it to guide token-by-token prediction. With four-token chunks, chunk token errors train its context forecast alongside next-token cross-entropy. In GPT2-XL experiments, it reports lower OpenWebText perplexity than NTP and four-token MTP. At 1.5B parameters, it uses 23% fewer training tokens and 20% fewer training FLOPs at comparable perplexity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.