Reproduce “GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures” on OpenResearch

About this paper

Jake R. Patock, Nicole Catherine Lewis, Kevin McCoy, Christina Gomez, Canling Chen, Lorenzo Luzi

GRR-CoCa, developed at Rice University, integrates modern architectural features from Large Language Models (LLMs) into the CoCa multimodal model, specifically updating its Vision Transformer (ViT) encoder and textual decoders. This enhanced model demonstrated consistent performance improvements in contrastive and generative image-text tasks, indicating that LLM mechanisms can benefit multimodal architectures without substantially increasing parameters.

Key insights

• Architectural mechanisms from LLMs (GEGLUs, RMSNorm, RoPe) can be effectively applied to multimodal models, including their visual encoders, leading to performance gains.
• Integrating RoPe throughout the Vision Transformer blocks likely helps preserve positional information more effectively in deeper layers, improving spatial understanding.
• GEGLUs and RMSNorm contribute to faster learning, better generalization, and more refined manipulation of embeddings by filtering noise and magnifying relevant features.

Summary by alphaXiv.