VRAM Bandwidth
Definition pending
We've cataloged "VRAM Bandwidth" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
VRAM bandwidth is the speed at which data moves from GPU memory to compute units. This is THE bottleneck for LLM inference. Every token generated requires reading ALL model weights from VRAM. For a 7B Q4 model (~4 GB), generating one token requires reading ~4 GB of data. At 1 TB/s bandwidth, that's 4ms per token = 250 tok/s (theoretical max). Real-world: ~50% of theoretical.
Workflow example
Calculate your model's speed limit: (1) model_size_GB = params × bytes_per_param (Q4=0.5, Q8=1, FP16=2), (2) theoretical_tok_s = vram_bandwidth_GB_s / model_size_GB, (3) real_tok_s ≈ theoretical_tok_s × 0.5. Example: 70B Q4 (~40 GB) on RTX 4090 (1 TB/s bandwidth): 1000/40 × 0.5 = ~12.5 tok/s. On H100 (3.35 TB/s): 3350/40 × 0.5 = ~42 tok/s. This formula is remarkably accurate — test it against ollama run --verbose.