Gemma 3 4B

4B Gemma 3 for edge. Multimodal.

Gemma 3 4B is a small multimodal model. Use a current Ollama release and check the Gemma 3 listing for the package and quantization. The command ollama pull gemma3:4b downloads the default 4B package; it does not by itself select a particular Q4_K_M artifact.

For llama.cpp, select a supported Gemma 3 GGUF and the matching vision projector when using images. The earlier example flags were -ngl 999 -fa -c 4096; verify their spelling against the installed release, and reduce GPU offload if memory is insufficient. Start with a modest context and check the allocated memory before increasing it.

A Q4 artifact is roughly 2.5–2.7 GB before runtime, vision and KV-cache memory, so a 2 GB device cannot hold the full working set. CPU-only use is possible with adequate system RAM, but no measured speed is linked here. The advertised maximum context does not guarantee that it fits a phone or small GPU.

For a larger model, see Gemma 3 12B. Review the Gemma license and upstream model card before deployment.

The Q4 artifact is approximately 2.5–2.7 GB before KV cache, vision components and runtime overhead. A 2 GB GPU or 2 GB system-RAM budget is insufficient for the full working set. Check the actual artifact and start with short context on a small device.

A larger GPU is optional for CPU inference with enough RAM. On Apple Silicon, memory is shared with macOS and other apps; confirm the runtime supports this model and its vision path. For phones, use a package and runtime explicitly supporting the device. No device-specific throughput measurement is linked here.

1. 4B quality ceiling. The model's fundamental capability is limited by 4B parameters. Complex reasoning, nuanced understanding, and deep knowledge recall hit the wall hard. 2. Hallucination rate. Small models hallucinate more. Gemma 3 4B will confidently produce incorrect facts more often than 7B+ models. Don't treat it as a knowledge base. 3. Context window overclaim. 8K is advertised but usable context at 4B degrades past 4K tokens. The model loses track of earlier context. Keep prompts concise. 4. Not multilingual. Gemma 3 4B's multilingual quality is weak. English is strongest; other languages may produce broken or English-mixed outputs.

Ollama for quick-start on any device. llama.cpp for CPU-only, Vulkan integrated GPU, or edge deployment. MLX-LM for Apple Silicon. MLC LLM for Android/iOS. Gemma 3 4B is universally supported — pick the runtime that matches your hardware. CPU-only via llama.cpp is surprisingly fast at this size.

Performance evidence

Performance statements in this entry have not been tied to a published, configuration-matched run. Treat speed figures as unverified estimates, not validated measurements. Compare the full configuration of a published benchmark before relying on throughput or latency.

Mistake: Deploying Gemma 3 4B for tasks that need reasoning depth. Fix: 4B is a lightweight model. For complex tasks, use at least Gemma 4 26B MoE or Qwen 3 32B. Mistake: Running 8K context and expecting coherent outputs. Fix: 4B quality degrades past 4K context. Keep prompts short and focused. Mistake: Using Gemma 3 4B as a production knowledge base. Fix: 4B models hallucinate more. Pair with RAG on vetted documents for factual tasks. Mistake: Over-provisioning hardware ("I need an RTX 4090 for this"). Fix: It runs on a phone. You don't need a powerful GPU. CPU-only inference at 10-20 tok/s is perfectly usable for chat.