Reproduce “Multimodal Latent Reasoning via Predictive Embeddings” on OpenResearch

About this paper

Ashutosh Adhikari, Mirella Lapata

PEARL learns embeddings of expert tool-assisted visual trajectories—intermediate images and reasoning text—from image-question pairs, then answers using ordinary vision-language decoding without tools or latent-token decoding. On Qwen2.5-VL-7B-Instruct trained with LVR data, it reaches 81.5% on V*, versus 80.1% for LVR and 79.1% for SFT. This comparison covers a visual-search benchmark with one tool type and one call.

Key insights

• Reconstruction-based latent reasoning methods primarily learn compact, useful embeddings rather than simulating dynamic visual transformations, as indicated by their insensitivity to the number of decoded latent tokens.
• Directly predicting latent representations of entire expert tool-use trajectories allows Vision-Language Models to internalize complex visual reasoning without explicit tool invocation at inference time.
• Exposure to diverse tool types and multi-step reasoning trajectories through specific training regimes can enhance model performance across varied perception-intensive tasks.

Summary by alphaXiv.