VRAM Bandwidth

VRAM bandwidth is the rate at which the GPU's video memory can transfer data to the compute cores, measured in GB/s. For local LLM inference, this determines how fast model weights and KV cache can be fed into the processing units. Higher bandwidth directly increases tokens-per-second, especially for large models or long contexts. It's a primary bottleneck after VRAM capacity: a GPU with 1 TB/s bandwidth can process a 7B model at ~50 tok/s, while one with 200 GB/s may manage only ~10 tok/s.

Bandwidth is often the limiting factor in LLM inference because the model weights must be read from VRAM for every token generated. The compute-to-bandwidth ratio means that even a GPU with many cores (e.g., RTX 4090 with 16,384 CUDA cores) will be underutilized if bandwidth is low. Memory bandwidth depends on memory type (GDDR6, GDDR6X, HBM2e, HBM3) and bus width. For example, an RTX 3090 has 936 GB/s (384-bit GDDR6X), while an RTX 4060 has 272 GB/s (128-bit GDDR6). Apple M-series chips use unified memory with very high bandwidth (e.g., M2 Ultra: 800 GB/s), which benefits large models that fit in system RAM. Operators should check bandwidth specs when choosing hardware for inference speed.

An RTX 4090 has 1,008 GB/s bandwidth and runs Llama 3.1 8B Q4 at 100 tok/s. An RTX 4060 with 272 GB/s runs the same model at ~30 tok/s. For a 70B Q4 model (40 GB), the RTX 4090 must offload to system RAM, dropping to 5 tok/s due to PCIe bandwidth (32 GB/s).

In llama.cpp, you can monitor bandwidth utilization with --verbose or via nvidia-smi. When running ./llama-cli -m model.gguf -n 512, if tokens/sec is lower than expected, check if VRAM bandwidth is saturated (GPU memory clock at max, memory controller utilization >90%). In LM Studio, the 'Performance' tab shows memory bandwidth usage during inference.