Reproduce “LightMTP: Lightweight Latent Multi-Token Prediction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tamara Czinczoll, Julie Kallini, Gerard de Melo, Chen Shani

LightMTP trains language models to predict their own future hidden-state vectors, adding latent future-token supervision without vocabulary projections for each prediction. Its auxiliary head uses stop-gradient targets and is removed at inference. On Countdown held-out targets, LightMTP-S at horizon k=8 achieves 56.7% strict accuracy versus NTP’s 46.4%; the result supports task-specific planning, not general reasoning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.