Reproduce “Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancement” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qimin Zhong, Hao Liao, Haiming Qin, Mingyang Zhou, Rui Mao, Wei Chen, Naipeng Chao

Researchers at Shenzhen University and Microsoft Research Asia developed Latent Semantic Enhancement Multi-Token Prediction (LSE-MTP) to improve the consistency of internal world models in sequence models. This method mitigates "structural hallucinations" by explicitly aligning latent predictions with ground-truth trajectories and semantic anchors, leading to more robust and coherent representations, particularly in structured reasoning tasks.

Key insights

  • Multi-Token Prediction (MTP) induces a gradient inductive bias that promotes representational contractivity, leading to the convergence of diverse histories into unified belief states.
  • Standard MTP exhibits "structural hallucinations," where it assigns probability to illegal shortcuts in the latent space to accurately predict distant tokens, compromising the physical validity of intermediate steps.
  • Anchoring multi-step latent predictions to ground-truth intermediate states and semantic embeddings is essential to prevent these hallucinations and ensure consistent internal dynamics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.