NVIDIA
NVIDIA designs the GPUs most operators use for local AI inference. Its consumer RTX series (e.g., RTX 4090) and workstation/enterprise cards (e.g., A6000, H100) provide the VRAM and compute throughput needed to run large language models. For local AI, the key specs are VRAM size (determines which model sizes fit) and memory bandwidth (determines tokens per second). NVIDIA's CUDA platform is the primary runtime for llama.cpp, vLLM, and Hugging Face Transformers on Windows/Linux.
Deeper dive
NVIDIA dominates local AI because its GPUs offer high VRAM capacities (8–80 GB) and fast memory bandwidth (up to 2 TB/s on H100). Consumer cards like the RTX 4090 (24 GB VRAM) can run 13B models at Q4 (8 GB) with room for context, while the RTX 3090 (24 GB) is a popular used option. Enterprise cards like the A100 (40/80 GB) or H100 (80 GB) are needed for 70B+ models without offloading. NVIDIA's CUDA cores and Tensor Cores accelerate matrix operations used in transformer inference. For local operators, NVIDIA GPUs are the most widely supported, with llama.cpp, Ollama, and LM Studio all offering CUDA backends. The main limitation is VRAM: a 16 GB card tops out around 13B Q4, while 70B Q4 (40 GB) requires a 48 GB card or system-RAM offload.
Practical example
An operator with an RTX 4090 (24 GB VRAM) can run Llama 3.1 70B at Q4_K_M (40 GB) only by offloading layers to system RAM, resulting in ~2 tok/s. The same card runs Llama 3.1 8B at Q4_K_M (5 GB) at ~80 tok/s. A 70B model fits entirely on an A100 80 GB, achieving ~30 tok/s.
Workflow example
When running llama-cli -m model.gguf -ngl 99 with an NVIDIA GPU, the -ngl flag offloads layers to VRAM. If VRAM is insufficient, the runtime falls back to CPU offload, and tokens/sec drops sharply. In LM Studio, the 'GPU Offload' slider controls the same behavior. Operators check VRAM usage with nvidia-smi to decide which quantized model fits.
Reviewed by Fredoline Eruo. See our editorial policy.