Custom build engine

Describe your build — any GPUs, CPU, RAM, OS, runtime, use case. We'll compute effective VRAM honestly, recommend a runtime, and tell you which models fit comfortably, which are borderline, and which aren't practical.

Total VRAM ≠ pooled VRAM. We never sum VRAM unless the silicon truly pools (Apple unified memory). We always explain why effective is lower than total.

Calculations follow the RunLocalAI Will-It-Run Framework: effective VRAM, model working set, runtime constraints, fit tiers, and measured-vs-estimated evidence labels.

Describe your build

Add GPUs, set CPU/RAM/OS, optionally pick a runtime + use case. URL updates as you change fields — share a build by copying the URL.

GPUs in your build

CPU class

System RAM (GB)

Use case

Your skill level

Runtime preference

Build summary

Total VRAM

320 GB

Effective VRAM

~265 GB

range 250-272 GB

Topology

single node multi gpu

pcie

Setup difficulty

advanced

speed penalty ~18%

Why effective VRAM is lower than total

4× NVIDIA H100 SXM = 320 GB total VRAM, but without NVLink, cross-card bandwidth is PCIe-bound (~32 GB/s vs NVLink ~112 GB/s). With tensor-parallelism, each card holds ~1/4 of the model weights and replicates activations + KV cache. After 15% TP overhead, effective model capacity is ~265 GB. Largest single tensor on one card is ~78 GB.

Measured evidence on this hardware

Publicly inspectable measured rows for the selected hardware slug(s). Exact measured rows calibrate the fit table instead of leaving it as pure VRAM estimation.

No publicly inspectable benchmark rows are attached to this exact hardware yet. The engine will still calculate fit and runtime, but speed rows will remain estimated.

Recommended runtime

Best engine for this topology + skill level + use case.

vLLM

primary

involved

Tensor-parallel across NVLink/PCIe — works on every recent consumer + datacenter pair. AWQ-INT4 + 70B fits dual 3090 / dual 4090 cleanly.

ExLlamaV2

alternative

involved

Single-stream king. EXL2 4.0bpw + 70B fits dual 3090 with NVLink and beats vLLM on solo-user throughput.

llama.cpp

alternative

moderate

Layer-split via --tensor-split is the experimentation-friendly path. Worse throughput than vLLM but easier to debug.

WORKLOAD PROFILE

FITS

Mixtral 8x22B Instruct @ Q4_K_M, 8K context on NVIDIA H100 SXM

0 GB265 GBVRAM ceiling

Weights84 GB

KV cache71 GB

Activations4.2 GB

Runtime1.8 GB

Headroom104 GB

ESTIMATED DECODE RATE

28 tok/s

Bandwidth-derived estimate · efficiency 0.70. Real-world rates land within ±20% on well-tuned runtimes.

Models that fit your build

177 models considered. Categorized by headroom at the recommended quant + a sensible context for your use case.

Comfortable

24 models · ≥15% headroom

Model	Params	Quant	VRAM est.	Context	Evidence	Note
Mixtral 8x22B Instruct	141B	Q4_K_M	158.7 GB	8,192	No measured row yet	Comfortable fit with 40% headroom — room to extend context or run alongside other workloads.
Mistral Large 2 (123B)	123B	Q4_K_M	138.2 GB	8,192	No measured row yet	Comfortable fit with 48% headroom — room to extend context or run alongside other workloads.
Llama 4 Scout	109B	Q5_K_M	136.4 GB	8,192	No measured row yet	Comfortable fit with 49% headroom — room to extend context or run alongside other workloads.
Command R+ (Aug 2024)	104B	AWQ-INT4	106.6 GB	8,192	No measured row yet	Comfortable fit with 60% headroom — room to extend context or run alongside other workloads.
Command R+ 104B	104B	Q4_K_M	115 GB	8,192	No measured row yet	Comfortable fit with 57% headroom — room to extend context or run alongside other workloads.
Llama 3.2 90B Vision	90B	AWQ-INT4	92.3 GB	8,192	No measured row yet	Comfortable fit with 65% headroom — room to extend context or run alongside other workloads.
Llama 3.2 90B Vision Instruct	90B	Q4_K_M	98.6 GB	8,192	No measured row yet	Comfortable fit with 63% headroom — room to extend context or run alongside other workloads.
InternVL 2.5 78B	78B	Q4_K_M	86.3 GB	8,192	No measured row yet	Comfortable fit with 67% headroom — room to extend context or run alongside other workloads.
Qwen 2.5 72B Instruct	72B	Q5_K_M	87.5 GB	8,192	No measured row yet	Comfortable fit with 67% headroom — room to extend context or run alongside other workloads.
Qwen 2.5 Math 72B	72B	Q4_K_M	61.1 GB	4,096	No measured row yet	Comfortable fit with 77% headroom — room to extend context or run alongside other workloads.
Qwen 2.5-VL 72B	72B	AWQ-INT4	80.1 GB	8,192	No measured row yet	Comfortable fit with 70% headroom — room to extend context or run alongside other workloads.
Molmo 72B	72B	BF16	172.4 GB	4,096	No measured row yet	Fits cleanly at BF16 + 4,096 ctx with 35% headroom.
DeepSeek R1 Distill Llama 70B	70B	Q5_K_M	84.4 GB	8,192	No measured row yet	Comfortable fit with 68% headroom — room to extend context or run alongside other workloads.
Llama 3.1 70B Instruct	70B	Q5_K_M	84.4 GB	8,192	No measured row yet	Comfortable fit with 68% headroom — room to extend context or run alongside other workloads.
Llama 3.3 70B Instruct	70B	Q8_0	76.2 GB	8,192	No measured row yet	Comfortable fit with 71% headroom — room to extend context or run alongside other workloads.
Hermes 3 Llama 3.1 70B	70B	Q4_K_M	77 GB	8,192	No measured row yet	Comfortable fit with 71% headroom — room to extend context or run alongside other workloads.
OpenBioLLM Llama 3 70B	70B	Q4_K_M	79.1 GB	8,192	No measured row yet	Comfortable fit with 70% headroom — room to extend context or run alongside other workloads.
Tulu 3 70B	70B	Q4_K_M	80.2 GB	8,192	No measured row yet	Comfortable fit with 70% headroom — room to extend context or run alongside other workloads.
Llama 3.1 Nemotron 70B Instruct	70B	Q4_K_M	79.6 GB	8,192	No measured row yet	Comfortable fit with 70% headroom — room to extend context or run alongside other workloads.
Jamba 1.5 Mini	52B	Q4_K_M	58.6 GB	8,192	No measured row yet	Comfortable fit with 78% headroom — room to extend context or run alongside other workloads.
Mixtral 8x7B Instruct	47B	Q5_K_M	58.2 GB	8,192	No measured row yet	Comfortable fit with 78% headroom — room to extend context or run alongside other workloads.
Command R 35B	35B	Q4_K_M	39.6 GB	8,192	No measured row yet	Comfortable fit with 85% headroom — room to extend context or run alongside other workloads.
Aya 23 35B	35B	Q4_K_M	39.6 GB	8,192	No measured row yet	Comfortable fit with 85% headroom — room to extend context or run alongside other workloads.
Yi 1.5 34B	34B	Q4_K_M	29.5 GB	4,096	No measured row yet	Comfortable fit with 89% headroom — room to extend context or run alongside other workloads.

Borderline

2 models · tight, may need quant downgrade

Model	Params	Quant	VRAM est.	Context	Evidence	Note
DeepSeek V2.5 236B	236B	Q4_K_M	258.7 GB	8,192	No measured row yet	Tight fit at Q4_K_M — only 2% headroom. KV cache for longer context will OOM. Cap context tighter or drop one quant level.
DeepSeek Coder V2 236B	236B	Q4_K_M	258.7 GB	8,192	No measured row yet	Tight fit at Q4_K_M — only 2% headroom. KV cache for longer context will OOM. Cap context tighter or drop one quant level.

Not practical

11 models · oversize for this build

Model	Params	Quant	VRAM est.	Context	Evidence	Note
Qwen 3 235B-A22B	235B	Q4_K_M	266.6 GB	8,192	No measured row yet	~266.6 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 1%. Drop quant or move to a larger build.
Llama 3.1 Nemotron Ultra 253B	253B	Q4_K_M	277.7 GB	8,192	No measured row yet	~277.7 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 5%. Drop quant or move to a larger build.
DeepSeek V4 Flash (284B MoE)	284B	Q4_K_M	312.1 GB	8,192	No measured row yet	~312.1 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 18%. Drop quant or move to a larger build.
Hunyuan Large 389B MoE	389B	Q3_K_M	393.6 GB	8,192	No measured row yet	~393.6 GB needed at Q3_K_M + 8,192 ctx — overshoots effective VRAM by 49%. Drop quant or move to a larger build.
Jamba 1.5 Large	398B	Q4_K_M	451 GB	8,192	No measured row yet	~451.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 70%. Drop quant or move to a larger build.
Llama 4 Maverick	400B	Q4_K_M	452 GB	8,192	No measured row yet	~452.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 71%. Drop quant or move to a larger build.
DeepSeek R1 (671B reasoning)	671B	Q4_K_M	734.5 GB	8,192	No measured row yet	~734.5 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 177%. Drop quant or move to a larger build.
DeepSeek V3 (671B MoE)	671B	Q4_K_M	734.5 GB	8,192	No measured row yet	~734.5 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 177%. Drop quant or move to a larger build.
Ring-2.6-1T	1000B	Q3_K_M	1011.9 GB	8,192	No measured row yet	~1011.9 GB needed at Q3_K_M + 8,192 ctx — overshoots effective VRAM by 282%. Drop quant or move to a larger build.
Kimi K2.6	1000B	Q4_K_M	1130 GB	8,192	No measured row yet	~1130.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 326%. Drop quant or move to a larger build.
DeepSeek V4 Pro (1.6T MoE)	1600B	Q4_K_M	1766 GB	8,192	No measured row yet	~1766.0 GB needed at Q4_K_M + 8,192 ctx — overshoots effective VRAM by 566%. Drop quant or move to a larger build.

Rent instead

Not enough VRAM for this model? Rent a GPU by the hour

This build can't hold Qwen 3 235B-A22B. By our estimate, a rented machine with at least 267 GB of GPU memory in total (at this size, usually several GPUs) runs it now, without buying new hardware.

Referral links: RunLocalAI may earn a commission or account credits if you sign up and rent, at no extra cost to you. Hourly rates and which GPUs are listed change all the time, so check the provider's page before you start. How we make money.

Multi-GPU buying guide →

NVLink vs PCIe, tensor- vs pipeline-parallel, mixed-card honesty.

Hardware combinations →

Curated multi-GPU / cluster setups with effective-VRAM math.

Setup path-finder →

OS + runtime install commands for your stack.

Compatibility matrix →

Runtime × OS × hardware support truth table.

Shopping a full build instead of a single card?

If you're sizing a fresh AI build (not just a card to drop into an existing system), the build-budget walkthroughs cover the whole BOM honestly: AI PC build under $1,000 or AI PC build under $2,000 cover the realistic 2026 budget tiers.

Vertical-fit shopping? AI PC for students covers the budget + portability tradeoffs; AI PC for developers covers the coding workflow specifics; AI PC for small business covers the document-RAG / always-on machine.

Form-factor first? See best laptop for local AI, best Mac for local AI, best mini PC for local AI, or best used GPU for local AI.