Single-node multi-GPUNVLinkintermediate

What runs on Dual RTX 3090 (24 GB × 2)?

The reference dual-GPU local-AI rig. NVLink optional. 48 GB total / ~46 GB effective with tensor parallelism. The cheapest path to 70B-class models at 2025-2026 prices.

At a glance
Effective VRAM
46 / 48 GB
Not pooled
Speed penalty
~10%
vs ideal single-card
Recommended runtime
vllm
tensor parallel
Setup difficulty
intermediate
~700W peak
24
Models fit
12
Borderline
8
Not practical
Deployment recipe
Dual RTX 3090 workstation →

Step-by-step setup with NVLink bridge verification, vLLM tensor-parallel-2 configuration, and operator-grade failure modes.

Memory budget
Total VRAM
48 GB
Effective for inference
46 GB
96% of total
Not pooled

PCIe + optional NVLink between two RTX 3090s does NOT pool VRAM the way Apple unified memory does. Each card holds its half of the model weights via tensor parallelism (vLLM / SGLang) or pipeline parallelism (llama.cpp layer split). Effective VRAM is roughly total minus ~2 GB per card for activations, KV cache, and runtime overhead. Concretely: a 70B Q4 model (~40 GB weights) fits with ~6 GB of headroom for context and KV. Anything claiming 48 GB pooled is wrong.

Why total VRAM is not the whole story

Two cards with NVLink bridge. NVLink (~112.5 GB/s bidirectional) keeps tensor-parallel efficient but does NOT pool memory. Each card holds its share via tensor or pipeline parallelism. Effective 46 GB of total 48 GB.

See the multi-GPU guide for topology tradeoffs, and the RunLocalAI Will-It-Run Framework for the citable fit-tier method.

Topology

Topology
single-node-multi-gpu
Interconnect
nvlink~112.5 GB/s
Component count
2 units
Components
Recommended runtime
vllm
Also: sglang, exllamav2, tgi
Recommended split strategy
tensor-parallel
Also: pipeline-parallel
Setup difficulty
intermediate
~700W peak

Models that fit comfortably (24)

Effective VRAM utilization ≤ 85% at the smallest production quant. Comfortable headroom for KV cache.

Jamba 1.5 Mini
Fits
52B·Q4_K_M → 36 GB·78% of effective VRAM·~10% speed penalty vs ideal
Mixtral 8x7B Instruct
Fits
47B·Q4_K_M → 32 GB·70% of effective VRAM·~10% speed penalty vs ideal
Aya 23 35B
Fits
35B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Command R 35B
Fits
35B·Q4_K_M → 26 GB·57% of effective VRAM·~10% speed penalty vs ideal
Phind CodeLlama 34B v2
Fits
34B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Yi 1.5 34B
Fits
34B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
DeepSeek R1 Distill Qwen 32B
Fits
32B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Aya Expanse 32B
Fits
32B·AWQ-INT4 → 22 GB·48% of effective VRAM·~10% speed penalty vs ideal
Qwen 2.5 32B Instruct
Fits
32B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Qwen 3 32B
Fits
32B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
QwQ 32B Preview
Fits
32B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Qwen 2.5 Coder 32B Instruct
Fits
32B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
EXAONE 3.5 32B
Fits
32B·AWQ-INT4 → 22 GB·48% of effective VRAM·~10% speed penalty vs ideal
Gemma 4 31B Dense
Fits
31B·Q4_K_M → 24 GB·52% of effective VRAM·~10% speed penalty vs ideal
Omni 31B Turkish Reasoning
Fits
31B·Q4_K_M → 22 GB·48% of effective VRAM·~10% speed penalty vs ideal
Qwen 3 30B-A3B
Fits
30B·Q4_K_M → 22 GB·48% of effective VRAM·~10% speed penalty vs ideal
Gemma 3 27B
Fits
27B·Q4_K_M → 20 GB·43% of effective VRAM·~10% speed penalty vs ideal
MedGemma 27B
Fits
27B·Q4_K_M → 20 GB·43% of effective VRAM·~10% speed penalty vs ideal
Qwen3.8 27B
Fits
27B·Q4_K_M → 20 GB·43% of effective VRAM·~10% speed penalty vs ideal
Gemma 4 Turkish 26B (4B active)
Fits
26B·Q4_K_M → 19 GB·41% of effective VRAM·~10% speed penalty vs ideal
InternVL 2.5 26B
Fits
26B·Q4_K_M → 20 GB·43% of effective VRAM·~10% speed penalty vs ideal
Mistral Small 3 24B
Fits
24B·Q4_K_M → 18 GB·39% of effective VRAM·~10% speed penalty vs ideal
Dolphin 3.0 Mistral 24B
Fits
24B·Q4_K_M → 18 GB·39% of effective VRAM·~10% speed penalty vs ideal
Codestral 22B
Fits
22B·Q4_K_M → 16 GB·35% of effective VRAM·~10% speed penalty vs ideal

Borderline (12)

Fits but with little headroom. KV cache for long context may not fit; verify before deployment.

InternVL 2.5 78B
Borderline
78B·Q4_K_M → 52 GB·113% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >113% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Qwen 2.5 72B Instruct
Borderline
72B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Qwen 2.5-VL 72B
Borderline
72B·AWQ-INT4 → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Molmo 72B
Borderline
72B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Qwen 2.5 Math 72B
Borderline
72B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Llama 3.1 70B Instruct
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Tulu 3 70B
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

DeepSeek R1 Distill Llama 70B
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Llama 3.1 Nemotron 70B Instruct
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

OpenBioLLM Llama 3 70B
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Llama 3.3 70B Instruct
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Hermes 3 Llama 3.1 70B
Borderline
70B·Q4_K_M → 48 GB·104% of effective VRAM·~10% speed penalty vs ideal

Effective VRAM utilization >104% — KV cache for long context will not fit. Cap context at ~4-8K or move to a larger combo.

Not practical (8)

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly. Drop to a smaller quant or move to a larger combo.

DeepSeek V4 Pro (1.6T MoE)
Not practical
1600B·Q4_K_M → 1024 GB·2226% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

Kimi K2.6
Not practical
1000B·Q4_K_M → 700 GB·1522% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

DeepSeek R1 (671B reasoning)
Not practical
671B·Q4_K_M → 420 GB·913% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

DeepSeek V3 (671B MoE)
Not practical
671B·Q4_K_M → 420 GB·913% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

Llama 4 Maverick
Not practical
400B·Q4_K_M → 280 GB·609% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

Jamba 1.5 Large
Not practical
398B·Q4_K_M → 260 GB·565% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

Hunyuan Large 389B MoE
Not practical
389B·Q4_K_M → 260 GB·565% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

DeepSeek V4 Flash (284B MoE)
Not practical
284B·Q4_K_M → 192 GB·417% of effective VRAM·~10% speed penalty vs ideal

Model weights exceed effective combo VRAM. Even with the recommended split strategy, this configuration won't run cleanly.

Benchmark opportunities

estimates, not measurements

Pending benchmark targets for this combo. Once measured, results land in the catalog as benchmarks.

Dual RTX 3090 + Llama 3.3 70B Q4 (vLLM tensor-parallel)
pending
Estimate: 25-32 tok/s decode (NVLink)

Reference benchmark for the dual-3090 NVLink prosumer build. vLLM tensor-parallel-2, AWQ-INT4, 8K context. Compare against dual-4090 PCIe (no NVLink) to isolate interconnect impact.

Going deeper