Reproduce “Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents” on OpenResearch

About this paper

XiuYu Zhang, Junfeng Fang, Zhenkai Liang

Researchers at the National University of Singapore demonstrate that cosine alignment, a common metric in Latent Visual Reasoning (LVR), negatively correlates with task accuracy in Vision-Language Models (VLMs), revealing that supervised latents are often bypassed. The study introduces PRISM, a two-axis diagnostic framework, which localizes where answer-relevant information is decodable and causally utilized within VLM architectures.

Key insights

• Auxiliary losses in VLMs primarily reshape the shared parameters of the language model through gradient flow, rather than directly improving the quality or utility of the nominally optimized latent variable.
• Supervised intermediate representations (latents) in LVR are frequently bypassed during inference, indicating that high fidelity to their targets does not guarantee their causal utilization for task performance.
• Answer-relevant information tends to migrate downstream from the explicitly supervised latents, residing in later computational stages of the VLM rather than being directly carried by the re-injected latent content.
• The "decodability gap" -- the difference in probe accuracy between the answer-decoding state and the feedback variable -- empirically predicts the model's reliance on its latents under perturbation.

Summary by alphaXiv.