Reproduce “TTRV: Test-Time Reinforcement Learning for Vision Language Models” on OpenResearch

About this paper

Akshit Singh, Shyam Marjit, Wei Lin, Paul Gavrikov, Serena Yeung-Levy, Hilde Kuehne, Rogerio Feris, Sivan Doveh and 2 more

TTRV introduces the first test-time reinforcement learning framework for decoder-based Vision-Language Models, enabling unsupervised adaptation on unlabeled test data by combining frequency-based and entropy-controlled self-supervised rewards with GRPO. This method yields substantial accuracy gains, averaging 28.97% on image recognition and 17.50% on VQA for InternVL3-8B, often outperforming GPT-4o.

Key insights

• Effective unsupervised adaptation of Vision-Language Models is achievable at test-time, enabling continuous model refinement without reliance on external labels.
• Latent visual recognition and reasoning capabilities within large pre-trained VLMs can be significantly activated and amplified through self-supervised reinforcement learning.
• A sophisticated self-supervised reward combining consistency (frequency) and certainty (low entropy) proves effective for guiding policy optimization in autoregressive multimodal models.

Summary by alphaXiv.