Phi-3.5 Mini Instruct
Compact 3.8B Phi for edge deployment. 128K context. Strong reasoning per parameter.
Positioning
The right pick when VRAM is the gating constraint — sub-6 GB cards, integrated GPUs, edge devices, or as a fast secondary model for routing/classification in agent loops. Microsoft's curation against synthetic textbooks shows: it's startlingly capable for 3.8B parameters.
Strengths
- 2.3 GB at Q4_K_M — runs on essentially anything that exists, including 4 GB GPUs with comfortable context.
- Structured output and math are genuinely good for the size class — better than Llama 3.2 3B on GSM8K and JSON-mode tasks.
- MIT license: cleanest license in the curated-data model space.
Limitations
- Open-domain knowledge is shallow — the textbook-only training shows on pop culture, recent events, and obscure technical lore.
- Refusal behavior is aggressive — defaults to over-cautious answers on anything dual-use.
- Long-context recall is weak despite the 128K spec — past ~16K, quality degrades sharply.
Real-world performance on RTX 4090
- Q4_K_M (2.3 GB): 130–155 tok/s decode, TTFT under 50 ms
- Q5_K_M (2.8 GB): 120–140 tok/s
- Q8_0 (4.1 GB): 100–120 tok/s — surprisingly worth it; Q8 quality bump is larger than usual
Should you run this locally?
Yes, for edge deployment, fast routing/classification in agent stacks, math-heavy structured tasks, or any rig with under 6 GB VRAM. No, for open-ended chat, creative writing, or current-events tasks.
How it compares
- vs Llama 3.2 3B → Phi wins on math + structured output; Llama wins on conversational naturalness and knowledge breadth. Pick Phi for tooling, Llama for chat.
- vs Llama 3.1 8B → Llama 3.1 8B is materially more capable across the board but uses 2× VRAM. Phi is the right pick only when VRAM matters.
- vs Gemma 3 4B → very close call; Gemma 3 4B has a slight edge on multilingual + general chat, Phi 3.5 Mini wins on math + JSON. Both excellent in the 4B class.
- vs Phi-4 14B → not in the same class; Phi-4 is competitive with Llama 3.1 8B, Phi-3.5 Mini is a different efficiency tier.
Run this yourself
ollama pull phi3.5:3.8b-mini-instruct-q4_K_M
ollama run phi3.5:3.8b-mini-instruct-q4_K_M
›Why this rating
7.2/10 — punches well above its parameter count, especially on math and structured output. Loses points to general models with 2× the params for general chat, but no other 4B-class model is in this league.
Overview
Compact 3.8B Phi for edge deployment. 128K context. Strong reasoning per parameter.
Featured in these stacks
The L3 execution stacks that pick this model as a recommended component, with the one-line note explaining the role it plays in each.
- Stack · L3·Homelab tier·Role: Alternative 3.8B model with stronger instruction-followingiPhone on-device AI stack — Llama 3.2 3B / Phi-3.5 Mini via MLX Swift
Phi-3.5 Mini is 3.8B and slightly heavier than Llama 3.2 3B but with better instruction-following polish. MIT licensed. Pick when prompt adherence matters more than raw throughput.
- Stack · L3·Homelab tier·Role: Primary 3.8B modelAndroid on-device AI stack — Phi-3.5 Mini / Llama 3.2 3B via MLC LLM or Qualcomm AI Hub
Phi-3.5 Mini at INT4 (~2.3GB) fits comfortably on 12GB+ Android phones. MIT licensed. Microsoft's published Phi Silica benchmarks demonstrate this exact configuration on Snapdragon X Elite.
Family & lineage
How this model relates to others in its lineage. Family members share architecture and training-data roots; parent / children edges record direct distillation or fine-tune relationships.
Strengths
- MIT license
- 128K context
- Edge-class footprint
Weaknesses
- Heavy refusals
- Synthetic-data quirks
Quantization variants
Each quantization trades model quality for file size and VRAM. Q4_K_M is the most popular starting point.
| Quantization | File size | VRAM required |
|---|---|---|
| Q4_K_M | 2.4 GB | 4 GB |
| Q8_0 | 4.1 GB | 5 GB |
Get the model
Ollama
One-line install
ollama run phi3.5:3.8bRead our Ollama review →HuggingFace
Original weights
Source repository — direct quantization required.
Benchmarks
Real measurements on real hardware. Numbers ship with the runner version, quant, and date.
| Hardware | Provenance | Quant | Ctx | Tokens / sec | TTFT | Date |
|---|---|---|---|---|---|---|
| NVIDIA GeForce RTX 3080 16GB (Mobile) | EditorialM | Q4_K_M | 4K | 155.4tok/s | 66 ms | Jun 2, 26 |
What to do next
Got this model running on real hardware? Share what you measured — the form arrives with the model pre-selected.
Hardware that runs this
Cards with enough VRAM for at least one quantization of Phi-3.5 Mini Instruct.
Models worth comparing
Same parameter band, plus what's one tier above and below — so you can decide what actually fits your hardware.
Frequently asked
What's the minimum VRAM to run Phi-3.5 Mini Instruct?
Can I use Phi-3.5 Mini Instruct commercially?
What's the context length of Phi-3.5 Mini Instruct?
How do I install Phi-3.5 Mini Instruct with Ollama?
Source: huggingface.co/microsoft/Phi-3.5-mini-instruct
Reviewed by RunLocalAI Editorial. See our editorial policy for how we research and verify model claims.
Related — keep moving
Verify Phi-3.5 Mini Instruct runs on your specific hardware before committing money.