Tight fit
Running Mistral Nemo 12B Instruct on NVIDIA GeForce RTX 3080 16GB (Mobile)
Mistral Nemo 12B Instruct fits at Q8_0, but headroom is tight (1 GB). Reduce context or use a smaller quant for safety.
Model size
12B params
Mistral Nemo 12B Instruct →Memory available
Recommended quant
Q8_0
Highest quality that fits
Quick start with Ollama
1. Install
ollama pull mistral-nemo:12b2. Run
ollama run mistral-nemo:12bDefault quant in Ollama is Q4_K_M. To use a different quant, append it: mistral-nemo:12b-q5_K_M.
Variants and what fits
| Quantization | File size | VRAM required | Fits on NVIDIA GeForce RTX 3080 16GB (Mobile)? |
|---|---|---|---|
| Q4_K_M | 7.5 GB | 9 GB | Yes |
| Q5_K_M | 8.7 GB | 11 GB | Yes |
| Q8_0 | 13.0 GB | 15 GB | Yes |
Real benchmarks
Frequently asked
Can NVIDIA GeForce RTX 3080 16GB (Mobile) run Mistral Nemo 12B Instruct?
Mistral Nemo 12B Instruct fits at Q8_0, but headroom is tight (1 GB). Reduce context or use a smaller quant for safety.
What quantization should I use?
Q8_0 is the highest-quality variant of Mistral Nemo 12B Instruct that fits in 16 GB VRAM. Lower-bit quants will be smaller but lose some quality.
How fast will it be?
Measured at 65.7 tok/s on this combination in our testing.
See also: Mistral Nemo 12B Instruct, NVIDIA GeForce RTX 3080 16GB (Mobile), all benchmarks.
Reviewed by RunLocalAI Editorial. See our editorial policy.