Reproduce “Gemma 3 Technical Report” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova and 208 more
Gemma 3 adds image understanding and longer contexts to open models. It uses five local attention layers per global layer and a 1,024-token window to reduce cache memory. Context limits are 128K tokens, except 1B at 32K. Preliminary human-rated Arena results give Gemma 3 27B IT 1,338 Elo versus Gemma 2 27B IT 1,220, measuring chat preference, not visual ability.
Key insights
- Five local attention layers between each global layer reduces KV-cache memory explosion
- Increased RoPE base frequency (10k to 1M) on global layers enables stable long-context processing
- Vision understanding can be achieved by treating images as sequences of soft tokens
- Strategic post-training improves specific capabilities without full retraining
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.