Reproduce “PaliGemma 2: A Family of Versatile VLMs for Transfer” on OpenResearch

About this paper

Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer and 10 more

PaliGemma 2, from Google DeepMind, introduces a family of open-weight Vision-Language Models built on the Gemma 2 language model and SigLIP vision encoder, prioritizing transfer learning for diverse downstream tasks. The models achieve state-of-the-art results across specialized domains such as OCR, table structure recognition, molecular chemistry, and medical imaging, while a systematic scaling analysis provides insights into optimizing performance with varying model sizes and image resolutions.

Key insights

• Tasks requiring fine-grained visual details (e.g., OCR, document analysis) benefit more from higher image resolution, while tasks needing complex reasoning or world knowledge gain more from larger language models.
• The 10B model often provides the most favorable balance of performance improvement for the computational cost, showing diminishing returns when scaling from 10B to 28B parameters.
• 8-bit quantization with `gemma.cpp` enables efficient CPU inference with virtually no quality degradation across various fine-tuning datasets, making the models accessible for resource-constrained environments.

Summary by alphaXiv.