Reproduce “JEPA-Reasoner: Decoupling Latent Reasoning from Token Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bingyang Kelvin Liu, Ziyu Patrick Chen, David P. Woodruff

JEPA-Reasoner separates continuous latent reasoning from text generation: a Reasoner completes an autoregressive latent trajectory, then a separate Talker reconstructs words without feeding sampled Talker tokens back into the Reasoner. On GSM8K, its approximately 0.9B-parameter system reached 51.9% accuracy versus 20.8% for a coupled Transformer, but this single comparison does not establish decoupling alone caused the gain.

Key insights

  • Decoupling reasoning from token generation mathematically prevents surface-level errors from propagating back into and corrupting the latent reasoning trajectory, leading to greater robustness.
  • The continuous nature of the latent space allows the Reasoner to maintain and operate on multiple hypotheses simultaneously via 'mixed latent vectors,' enabling a foundation for multi-threaded reasoning.
  • The Talker's complete reliance on the Reasoner's output ensures that linguistic expression is always guided by a coherent and fully formed high-level reasoning plan.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.