Custom build engine

Describe your build — any GPUs, CPU, RAM, OS, runtime, use case. We'll compute effective VRAM honestly, recommend a runtime, and tell you which models fit comfortably, which are borderline, and which aren't practical.

Total VRAM ≠ pooled VRAM. We never sum VRAM unless the silicon truly pools (Apple unified memory). We always explain why effective is lower than total.

Calculations follow the RunLocalAI Will-It-Run Framework: effective VRAM, model working set, runtime constraints, fit tiers, and measured-vs-estimated evidence labels.

Describe your build

Add GPUs, set CPU/RAM/OS, optionally pick a runtime + use case. URL updates as you change fields — share a build by copying the URL.

GPUs in your build

CPU class

System RAM (GB)

Use case

Your skill level

Runtime preference

Build summary

Total VRAM

96 GB

Effective VRAM

~75 GB

range 71-77 GB

Topology

single node multi gpu

pcie

Setup difficulty

advanced

speed penalty ~18%

Why effective VRAM is lower than total

4× NVIDIA GeForce RTX 3090 = 96 GB total VRAM, but without NVLink, cross-card bandwidth is PCIe-bound (~32 GB/s vs NVLink ~112 GB/s). With tensor-parallelism, each card holds ~1/4 of the model weights and replicates activations + KV cache. After 15% TP overhead, effective model capacity is ~75 GB. Largest single tensor on one card is ~22 GB.

Measured evidence on this hardware

Publicly inspectable measured rows for the selected hardware slug(s). Exact measured rows calibrate the fit table instead of leaving it as pure VRAM estimation.

No publicly inspectable benchmark rows are attached to this exact hardware yet. The engine will still calculate fit and runtime, but speed rows will remain estimated.

Recommended runtime

Best engine for this topology + skill level + use case.

vLLM

primary

involved

Tensor-parallel across NVLink/PCIe — works on every recent consumer + datacenter pair. AWQ-INT4 + 70B fits dual 3090 / dual 4090 cleanly.

ExLlamaV2

alternative

involved

Single-stream king. EXL2 4.0bpw + 70B fits dual 3090 with NVLink and beats vLLM on solo-user throughput.

llama.cpp

alternative

moderate

Layer-split via --tensor-split is the experimentation-friendly path. Worse throughput than vLLM but easier to debug.

WORKLOAD PROFILE

FITS

Qwen 2.5 Math 72B @ Q4_K_M, 4K context on NVIDIA GeForce RTX 3090

0 GB75 GBVRAM ceiling

Weights41 GB

KV cache18 GB

Activations2.1 GB

Runtime1.8 GB

Headroom12 GB

ESTIMATED DECODE RATE

16 tok/s

Bandwidth-derived estimate · efficiency 0.70. Real-world rates land within ±20% on well-tuned runtimes.

Models that fit your build

177 models considered. Categorized by headroom at the recommended quant + a sensible context for your use case.

Comfortable

24 models · ≥15% headroom

Model	Params	Quant	VRAM est.	Context	Evidence	Note
Qwen 2.5 Math 72B	72B	Q4_K_M	61.1 GB	4,096	No measured row yet	Fits cleanly at Q4_K_M + 4,096 ctx with 19% headroom.
Llama 3.3 70B Instruct	70B	Q5_K_M	52 GB	8,192	No measured row yet	Fits cleanly at Q5_K_M + 8,192 ctx with 31% headroom.
Jamba 1.5 Mini	52B	Q4_K_M	58.6 GB	8,192	No measured row yet	Fits cleanly at Q4_K_M + 8,192 ctx with 22% headroom.
Mixtral 8x7B Instruct	47B	Q5_K_M	58.2 GB	8,192	No measured row yet	Fits cleanly at Q5_K_M + 8,192 ctx with 22% headroom.
Command R 35B	35B	Q4_K_M	39.6 GB	8,192	No measured row yet	Comfortable fit with 47% headroom — room to extend context or run alongside other workloads.
Aya 23 35B	35B	Q4_K_M	39.6 GB	8,192	No measured row yet	Comfortable fit with 47% headroom — room to extend context or run alongside other workloads.
Yi 1.5 34B	34B	Q4_K_M	29.5 GB	4,096	No measured row yet	Comfortable fit with 61% headroom — room to extend context or run alongside other workloads.
Phind CodeLlama 34B v2	34B	Q4_K_M	38 GB	8,192	No measured row yet	Comfortable fit with 49% headroom — room to extend context or run alongside other workloads.
Aya Expanse 32B	32B	AWQ-INT4	32.8 GB	8,192	No measured row yet	Comfortable fit with 56% headroom — room to extend context or run alongside other workloads.
Qwen 2.5 32B Instruct	32B	Q8_0	51.7 GB	8,192	No measured row yet	Fits cleanly at Q8_0 + 8,192 ctx with 31% headroom.
Qwen 2.5 Coder 32B Instruct	32B	Q8_0	37.9 GB	8,192	No measured row yet	Comfortable fit with 50% headroom — room to extend context or run alongside other workloads.
Qwen 3 32B	32B	Q8_0	51.7 GB	8,192	No measured row yet	Fits cleanly at Q8_0 + 8,192 ctx with 31% headroom.
QwQ 32B Preview	32B	Q4_K_M	36 GB	8,192	No measured row yet	Comfortable fit with 52% headroom — room to extend context or run alongside other workloads.
DeepSeek R1 Distill Qwen 32B	32B	Q8_0	51.7 GB	8,192	No measured row yet	Fits cleanly at Q8_0 + 8,192 ctx with 31% headroom.
EXAONE 3.5 32B	32B	AWQ-INT4	36 GB	8,192	No measured row yet	Comfortable fit with 52% headroom — room to extend context or run alongside other workloads.
Gemma 4 31B Dense	31B	Q8_0	50.2 GB	8,192	No measured row yet	Fits cleanly at Q8_0 + 8,192 ctx with 33% headroom.
Omni 31B Turkish Reasoning	31B	Q4_K_M	33.5 GB	8,192	No measured row yet	Comfortable fit with 55% headroom — room to extend context or run alongside other workloads.
Qwen 3 30B-A3B	30B	Q8_0	48.6 GB	8,192	No measured row yet	Fits cleanly at Q8_0 + 8,192 ctx with 35% headroom.
Qwen3.8 27B	27B	Q8_0	44 GB	8,192	No measured row yet	Comfortable fit with 41% headroom — room to extend context or run alongside other workloads.
Gemma 3 27B	27B	Q8_0	44 GB	8,192	No measured row yet	Comfortable fit with 41% headroom — room to extend context or run alongside other workloads.
MedGemma 27B	27B	Q4_K_M	30.3 GB	8,192	No measured row yet	Comfortable fit with 60% headroom — room to extend context or run alongside other workloads.
InternVL 2.5 26B	26B	Q4_K_M	29.8 GB	8,192	No measured row yet	Comfortable fit with 60% headroom — room to extend context or run alongside other workloads.
Mistral Small 3 24B	24B	Q8_0	39.3 GB	8,192	No measured row yet	Comfortable fit with 48% headroom — room to extend context or run alongside other workloads.
Dolphin 3.0 Mistral 24B	24B	Q4_K_M	26.7 GB	8,192	No measured row yet	Comfortable fit with 64% headroom — room to extend context or run alongside other workloads.

Borderline

1 models · tight, may need quant downgrade

Model	Params	Quant	VRAM est.	Context	Evidence	Note
Qwen 3.6 27B (MTP)	27B	FP16	70.2 GB	8,192	No measured row yet	Tight fit at FP16 — only 6% headroom. KV cache for longer context will OOM. Cap context tighter or drop one quant level.

Not practical

16 models · oversize for this build

Model	Params	Quant	VRAM est.	Context	Evidence	Note
DeepSeek R1 Distill Llama 70B	70B	Q4_K_M	77 GB	8,192	No measured row yet	~77.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 3%. Drop quant or move to a larger build.
Llama 3.1 70B Instruct	70B	Q4_K_M	77 GB	8,192	No measured row yet	~77.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 3%. Drop quant or move to a larger build.
Hermes 3 Llama 3.1 70B	70B	Q4_K_M	77 GB	8,192	No measured row yet	~77.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 3%. Drop quant or move to a larger build.
OpenBioLLM Llama 3 70B	70B	Q4_K_M	79.1 GB	8,192	No measured row yet	~79.1 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 5%. Drop quant or move to a larger build.
Tulu 3 70B	70B	Q4_K_M	80.2 GB	8,192	No measured row yet	~80.2 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 7%. Drop quant or move to a larger build.
Llama 3.1 Nemotron 70B Instruct	70B	Q4_K_M	79.6 GB	8,192	No measured row yet	~79.6 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 6%. Drop quant or move to a larger build.
Qwen 2.5 72B Instruct	72B	Q4_K_M	79.1 GB	8,192	No measured row yet	~79.1 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 5%. Drop quant or move to a larger build.
Qwen 2.5-VL 72B	72B	AWQ-INT4	80.1 GB	8,192	No measured row yet	~80.1 GB needed at AWQ-INT4 + 8,192 ctx — overshoots effective VRAM by 7%. Drop quant or move to a larger build.
Molmo 72B	72B	F32	325.7 GB	4,096	No measured row yet	~325.7 GB needed at F32 + 4,096 ctx — overshoots effective VRAM by 334%. Drop quant or move to a larger build.
InternVL 2.5 78B	78B	Q4_K_M	86.3 GB	8,192	No measured row yet	~86.3 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 15%. Drop quant or move to a larger build.
Llama 3.2 90B Vision	90B	AWQ-INT4	92.3 GB	8,192	No measured row yet	~92.3 GB needed at AWQ-INT4 + 8,192 ctx — overshoots effective VRAM by 23%. Drop quant or move to a larger build.
Llama 3.2 90B Vision Instruct	90B	Q4_K_M	98.6 GB	8,192	No measured row yet	~98.6 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 31%. Drop quant or move to a larger build.
Command R+ (Aug 2024)	104B	AWQ-INT4	106.6 GB	8,192	No measured row yet	~106.6 GB needed at AWQ-INT4 + 8,192 ctx — overshoots effective VRAM by 42%. Drop quant or move to a larger build.
Command R+ 104B	104B	Q4_K_M	115 GB	8,192	No measured row yet	~115.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 53%. Drop quant or move to a larger build.
Llama 4 Scout	109B	Q4_K_M	122.8 GB	8,192	No measured row yet	~122.8 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 64%. Drop quant or move to a larger build.
Mistral Large 2 (123B)	123B	Q4_K_M	138.2 GB	8,192	No measured row yet	~138.2 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 84%. Drop quant or move to a larger build.

Rent instead

Not enough VRAM for this model? Rent a GPU by the hour

This build can't hold DeepSeek R1 Distill Llama 70B. By our estimate, a rented GPU with at least 77 GB runs it now, without buying new hardware.

Referral links: RunLocalAI may earn a commission or account credits if you sign up and rent, at no extra cost to you. Hourly rates and which GPUs are listed change all the time, so check the provider's page before you start. How we make money.

Multi-GPU buying guide →

NVLink vs PCIe, tensor- vs pipeline-parallel, mixed-card honesty.

Hardware combinations →

Curated multi-GPU / cluster setups with effective-VRAM math.

Setup path-finder →

OS + runtime install commands for your stack.

Compatibility matrix →

Runtime × OS × hardware support truth table.

Shopping a full build instead of a single card?

If you're sizing a fresh AI build (not just a card to drop into an existing system), the build-budget walkthroughs cover the whole BOM honestly: AI PC build under $1,000 or AI PC build under $2,000 cover the realistic 2026 budget tiers.

Vertical-fit shopping? AI PC for students covers the budget + portability tradeoffs; AI PC for developers covers the coding workflow specifics; AI PC for small business covers the document-RAG / always-on machine.

Form-factor first? See best laptop for local AI, best Mac for local AI, best mini PC for local AI, or best used GPU for local AI.