Reproduce “CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion” on OpenResearch

About this paper

Moritz Böhle, Amélie Royer, Juliette Marrie, Edouard Grave, Patrick Pérez

Researchers at Kyutai introduce CASA, a re-evaluation of cross-attention for Vision-Language Models, demonstrating that cross-attention can achieve performance within approximately 1.5% of token-insertion methods while offering substantial efficiency gains, particularly for streaming applications where it maintains low, constant memory usage and latency.

Key insights

• Modern training pipelines and controlled experimental settings can close the performance gap between cross-attention and token-insertion, with vanilla CA performing within 1.5% of token-insertion on image benchmarks.
• Cross-attention offers inherent efficiency advantages, maintaining low and constant memory and latency for streaming multimodal inputs, a critical feature token-insertion models lack.
• Architectural variants of cross-attention, such as parameter sharing or interleaved attention layers, allow for flexible trade-offs between efficiency and performance, enabling adaptation to specific computational budgets.

Summary by alphaXiv.