Gemma 3 4B

Gemma 3 4B is Google's smallest dense model — 4B parameters, designed for the "runs on literally anything" tier. Run at Q4_K_M via Ollama (ollama pull gemma3:4b) or llama.cpp with -ngl 999 -fa -c 4096. Q4_K_M file size ~2.5 GB on disk. Minimum VRAM: 2 GB — integrated graphics, phone GPU, or any discrete GPU. CPU-only works fine at 10-20 tok/s on modern laptop. Recommended: any device with 4+ GB RAM. Throughput: ~100-150+ tok/s on RTX 4090 at Q4_K_M — flies. ~15-25 tok/s CPU-only on MacBook Air. Gemma 3 architecture — well-supported everywhere. The 4B is the tiny-but-capable tier: good for text classification, extraction, simple Q&A, grammar correction, lightweight chat. Not for: complex reasoning, long-form generation, nuanced conversation, coding — 4B is the quality ceiling. Context: 8K advertised; practical at Q4 is 8K on any device. For step-up: Gemma 3 12B or Granite 3 MoE 3B-Active for MoE at similar active size with better quality. License: Gemma license (permissive, verify commercial terms). Gemma 3 4B is the default recommendation for edge/mobile/CPU-only when quality requirements are modest and speed + deployability matter most.

Minimum: 2 GB RAM CPU-only at Q4_K_M (~5-10 tok/s). Raspberry Pi 5 8GB: Q4 at 10-18 tok/s. Phone: runs via MLC LLM or llama.cpp Android at 5-10 tok/s. VRAM math: 4B dense, Q4_K_M ≈ 2.5 GB. KV cache at 4K: ~0.5 GB. Total: ~3 GB at 4K. Any discrete GPU from GTX 1050 upward fits comfortably. RTX 4090 24GB: absurdly over-provisioned — but delivers 150+ tok/s. CPU-only on modern laptop (16 GB RAM): 10-20 tok/s. Intel integrated graphics: 8-15 tok/s via llama.cpp Vulkan backend. MacBook Air M1 8GB: Q4 at 15-25 tok/s via MLX-LM. This is the most deployable model — runs on phones, Raspberry Pi, old laptops, AWS t3.micro. The hardware floor is essentially "any device made after 2018."

1. 4B quality ceiling. The model's fundamental capability is limited by 4B parameters. Complex reasoning, nuanced understanding, and deep knowledge recall hit the wall hard. 2. Hallucination rate. Small models hallucinate more. Gemma 3 4B will confidently produce incorrect facts more often than 7B+ models. Don't treat it as a knowledge base. 3. Context window overclaim. 8K is advertised but usable context at 4B degrades past 4K tokens. The model loses track of earlier context. Keep prompts concise. 4. Not multilingual. Gemma 3 4B's multilingual quality is weak. English is strongest; other languages may produce broken or English-mixed outputs.

Ollama for quick-start on any device. llama.cpp for CPU-only, Vulkan integrated GPU, or edge deployment. MLX-LM for Apple Silicon. MLC LLM for Android/iOS. Gemma 3 4B is universally supported — pick the runtime that matches your hardware. CPU-only via llama.cpp is surprisingly fast at this size.

Mistake: Deploying Gemma 3 4B for tasks that need reasoning depth. Fix: 4B is a lightweight model. For complex tasks, use at least Gemma 4 26B MoE or Qwen 3 32B. Mistake: Running 8K context and expecting coherent outputs. Fix: 4B quality degrades past 4K context. Keep prompts short and focused. Mistake: Using Gemma 3 4B as a production knowledge base. Fix: 4B models hallucinate more. Pair with RAG on vetted documents for factual tasks. Mistake: Over-provisioning hardware ("I need an RTX 4090 for this"). Fix: It runs on a phone. You don't need a powerful GPU. CPU-only inference at 10-20 tok/s is perfectly usable for chat.