vLLM

What this tool actually is

vLLM is the high-throughput inference engine that turned self-hosted LLM serving from a research project into a production category. Calling it "an OpenAI-compatible server" — which is how vendor docs and most listings frame it — undersells it by one whole architectural layer. The OpenAI compatibility is the wrapper; the engine underneath is a paged-memory KV cache scheduler with continuous batching that gets ~5-24x more throughput than naïve HuggingFace generate() on the same hardware.

The layer it occupies in the stack:

• Below: the model weights (HuggingFace format, AWQ, GPTQ, FP8, BNB) on one or more GPUs. CUDA / ROCm / TPU / Gaudi backends.
• Above: any HTTP client that speaks the OpenAI Chat / Completions API — or, increasingly, a Kubernetes-orchestrated serving layer like Ray Serve or KServe.

What it replaces: HuggingFace TGI for teams that want raw throughput over polish, hand-rolled FastAPI + transformers wrappers, and the "we'll just call OpenAI" line item once the math stops working. The 2025-2026 cycle made vLLM the assumed answer when an engineering team says "we need an LLM endpoint."

Who it is for. Anyone serving more than ~1 request/second sustained, anyone who needs multi-tenancy on a single endpoint, anyone running 70B+ models that need tensor parallelism. Who it is not for. Single-user laptop chat (use Ollama), Apple Silicon (use MLX-LM), edge / mobile deployment (use llama.cpp), or NVIDIA-only shops that need every last token-per-second on Hopper / Blackwell (compile to TensorRT-LLM instead).

Architecture

The mental model that makes vLLM make sense — and that explains why its throughput numbers look impossible until you understand the memory layout:

┌────────────────────────────────────────────────────────────────┐
│  vLLM AsyncLLMEngine                                           │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Scheduler                                               │  │
│  │   - prefill queue, decode queue                          │  │
│  │   - continuous batching: new request joins on next step  │  │
│  │   - chunked prefill: long prompts split, interleaved     │  │
│  └─────────────────────────┬────────────────────────────────┘  │
│                            │                                    │
│  ┌─────────────────────────▼────────────────────────────────┐  │
│  │  PagedAttention KV-cache manager                         │  │
│  │   - GPU memory split into fixed-size blocks (e.g. 16t)   │  │
│  │   - per-request "page table" maps logical→physical block │  │
│  │   - prefix cache: shared system prompts stay resident    │  │
│  └─────────────────────────┬────────────────────────────────┘  │
│                            │                                    │
│  ┌─────────────────────────▼────────────────────────────────┐  │
│  │  Worker(s) — one per GPU                                 │  │
│  │   - tensor parallel within a node (NCCL all-reduce)      │  │
│  │   - pipeline parallel across nodes (Ray transports)      │  │
│  │   - speculative decoding (draft + target model)          │  │
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘

Three things to understand:

1. PagedAttention is the headline innovation. Naïve KV-cache allocates contiguous memory per request sized to the max sequence length — wasting 60-80% of cache space when most requests are short. PagedAttention chops cache into fixed blocks (default 16 tokens) and assigns blocks dynamically. Memory efficiency goes from ~30% to >95%, and that headroom turns directly into batch size, which turns into throughput.
2. Continuous batching runs requests through prefill and decode independently. New requests join the running batch on the next decode step rather than waiting for the current batch to finish. This is what closes the latency gap with single-request inference at high QPS.
3. Prefix caching matters more than benchmarks suggest. When 100 requests share the same system prompt, vLLM keeps that prefix's KV cache resident across requests — first-token latency drops from 200ms to 10ms for cache hits. Multi-tenant chat applications get most of vLLM's wall-clock win from this single feature.

The serving layer on top is a thin FastAPI wrapper that exposes /v1/chat/completions, /v1/completions, and /v1/embeddings. Same client SDKs that speak OpenAI work without modification.

Local stack compatibility

vLLM is GPU-and-Linux-biased by design. CUDA is the mature backend; ROCm is good and getting better; Gaudi / TPU / Metal are partial-to-experimental. The matrix above shows nine backends with the operational notes that matter when wiring each. The short version: NVIDIA H100/A100/L40S are reference targets, RTX 4090/5090 are fine for single-card homelab, MI300X is the credible AMD alternative for 192GB-HBM workloads, and everything else needs verification on your specific model. Apple Silicon and CPU paths exist but you'd be using vLLM wrong on either.

Real deployment paths

The four ways teams actually run vLLM in 2026, ordered by operator skill required. (Cards above this section show hardware + complexity at a glance; the prose here is operator-grade detail.)

The single-GPU homelab path is where most readers start. pip install vllm, then vllm serve meta-llama/Llama-3.1-8B-Instruct, point any OpenAI client at http://localhost:8000/v1. Total time from zero to working endpoint is under 10 minutes if CUDA is already set up. The constraint is KV cache headroom: a 13B FP16 model on a 24GB card leaves ~6-8GB for KV, which caps you at roughly 32K total tokens in flight. Drop to AWQ-INT4 to triple that.

The multi-GPU server path is the production middle. tensor_parallel_size=4 shards a 70B model across 4xA100 80GB, with NCCL all-reduce on every layer. The key constraint people miss: NVLink matters. PCIe-only multi-GPU works but loses 30-40% of throughput to interconnect bandwidth. Pipeline parallelism is rarely worth it within a node — use TP unless the model literally won't fit.

The multi-node distributed path is where vLLM's design starts to look like Pathways or DeepSpeed-Inference. Ray orchestrates one head node + N worker nodes; tensor parallel within each node, pipeline parallel across nodes. Required for 405B / 671B-class models. InfiniBand or RoCE is non-negotiable; Ethernet kills you on inter-stage activations. Expect to spend a week on Ray cluster sizing before throughput numbers stabilize.

The Kubernetes production path is where serious self-hosted inference lives. KServe or Ray Serve in front of vLLM, GPU operator below, an autoscaler that spins up replicas based on QPS or queue depth. This is what teams build when they replace an OpenAI bill — the engineering investment is real (1-2 platform engineers' worth of attention) but the ROI math works at $50K+/month OpenAI spend.

Resource usage and performance

Numbers to plan around (single-card unless noted):

• VRAM = model weights + KV cache + activations + overhead. Rough budget: 70B FP16 ≈ 140GB weights, so it doesn't fit on a single 80GB card — TP across 2x H100 with ~15GB each left for KV. 70B AWQ-INT4 ≈ 35-40GB weights and fits on one H100 with 30GB+ KV headroom.
• gpu_memory_utilization (default 0.9) is the most important knob you'll touch. Lower it to 0.85 if you OOM on the first request; raise it to 0.95 if you have CPU headroom and want bigger batches. Do not touch max_num_seqs until you've understood this one.
• Throughput math: on H100, Llama-3.1-70B AWQ at concurrent batch ~32 sustains ~3500 tok/s aggregate. On 4xA100 with TP, the same workload runs ~2800 tok/s — the H100 wins on memory bandwidth.
• TTFT (time to first token): ~50ms cold prefix on a short prompt; <10ms warm prefix cache. Long-context (32K+ prompt) prefills take 1-3 seconds — chunked prefill smooths this out by interleaving prefill and decode.
• Prefix cache hit rate is the metric that predicts your wall-clock cost. Workloads with shared system prompts (chat apps, RAG with stable instructions) hit 70-90%; ad-hoc generation hits ~5%. Plan capacity at the workload's actual hit rate.

The honest scaling limit on a single replica: ~50-100 concurrent active requests before scheduler tail latency degrades. Past that, scale horizontally — more replicas behind a router beats one mega-replica.

Failure modes

The list of things that will go wrong in production, in rough order of how often we've seen them:

1. OOM on first inference (not on load). Model loads fine, first request crashes with CUDA OOM. The KV cache wasn't sized correctly. Lower gpu_memory_utilization from 0.9 to 0.85, or set max_model_len lower than the model's native context. vLLM doesn't reserve KV up front aggressively enough for some configs.
2. tensor_parallel_size mismatched to GPU count. Setting TP=4 on an 8-GPU box leaves 4 cards idle and silently halves throughput. Always verify with nvidia-smi that all expected GPUs see traffic.
3. Long-prompt prefill spike stalls all decode requests. A 32K-token prompt without chunked prefill enabled blocks the whole engine for 1-3 seconds. Enable --enable-chunked-prefill in any deployment that accepts long contexts.
4. Prefix cache invalidation on system prompt drift. Changing the system prompt by one token invalidates the cached prefix. Apps that template variable user data into the system prompt get 0% cache hit rate; move variable parts to the user message.
5. Driver / CUDA / PyTorch version mismatch. vLLM ships against specific CUDA + PyTorch versions per release. Mismatch causes silent kernel selection failures (you don't get errors, you get 30% throughput). Always pin the Docker image rather than pip install vllm on the host.
6. Streaming connection drop with no client cleanup. If a client disconnects mid-stream, vLLM keeps generating until max_tokens then drops the result. Set tight max_tokens and use abort signals; otherwise GPU time burns on dead requests.
7. Multi-LoRA loading silent failure. Loading 8 LoRAs against a base model — vLLM applies them but the wrong adapter binds to the wrong request under high concurrency. Fixed in 0.6+ but worth verifying with synthetic traffic.
8. NCCL hang on multi-node TP. Inter-node tensor parallel with non-uniform NIC settings deadlocks at startup. NCCL_DEBUG=INFO and NCCL_IB_HCA configuration are mandatory; assume a half-day of cluster debugging the first time.

How it compares

vs SGLang. SGLang is the credible challenger as of 2026. RadixAttention vs PagedAttention is the architectural difference: SGLang's tree-structured cache is faster on heavily-shared prefix workloads (think structured generation, agent loops with stable system prompts); vLLM's flat block model is faster on diverse-prompt workloads. Pick SGLang if your traffic is structurally repetitive; vLLM if it isn't.

vs TensorRT-LLM. TensorRT-LLM compiles a model to a fixed engine for a specific GPU; vLLM runs PyTorch kernels with dynamic batching. TensorRT-LLM wins on raw single-request latency on Hopper / Blackwell (15-30% lower TTFT). vLLM wins on iteration speed (no recompile to test a config change), backend coverage, and quant flexibility. Use TensorRT-LLM when you've committed to one SKU and need every microsecond.

vs HuggingFace TGI. TGI was the production default in 2023-2024; vLLM ate that lunch through 2024-2025. TGI still has tighter HF Hub integration and slightly nicer ops surface; vLLM has the throughput lead and the ecosystem momentum. New deployments default to vLLM unless HF integration matters.

vs llama.cpp server mode. llama.cpp is the right answer for CPU, Apple Silicon, edge, and "I want it to work on a Raspberry Pi." vLLM is the right answer for GPU production scale. Different categories — they barely overlap.

vs Ray Serve. Not an alternative — a layer above. Ray Serve orchestrates vLLM (or other engines) into multi-replica autoscaling deployments. The right pattern is Ray Serve + vLLM, not Ray Serve vs vLLM.

Best use cases

Where vLLM is genuinely the right answer:

• Self-hosted OpenAI-API replacement at SaaS scale. Same client code, different bill.
• Multi-tenant inference behind a single endpoint — prefix caching makes the per-request cost collapse on shared system prompts.
• 70B+ model serving that needs tensor parallelism. The TP path is mature and predictable.
• High-throughput batch inference — process millions of prompts overnight, vLLM's continuous batching turns it into hours instead of days.
• Multi-LoRA serving where one base model + many adapters serves multiple fine-tuned variants from the same memory.

Where vLLM is the wrong answer:

• Single-user laptop chat (Ollama is simpler).
• Apple Silicon (MLX-LM).
• Edge / mobile / Raspberry Pi (llama.cpp).
• Hard real-time, single-request, NVIDIA-only (TensorRT-LLM).
• Models with attention variants vLLM hasn't merged kernels for (verify before committing).

Verdict

vLLM is the production-default inference engine for self-hosted LLM serving in 2026. PagedAttention turned KV-cache memory efficiency from a research footnote into a 5-24x throughput delta against naïve baselines, and the project's discipline through 2024-2025 — continuous batching, prefix caching, chunked prefill, multi-LoRA, speculative decoding — turned that single innovation into a complete production stack. The OpenAI-compatible API on top makes it a drop-in for any team running an OpenAI bill they'd rather not pay.

The honest tradeoffs: it's GPU-and-Linux-biased; the operations skill required is real (NCCL, gpu_memory_utilization, prefix cache invalidation are not "configure once and forget" knobs); and on heavily structured workloads SGLang now has a credible architectural advantage. None of those are reasons to default away from vLLM — they're the conditions under which a different tool might win.

Buy / use this if you're serving more than ~1 request/second sustained on GPU and you don't have a hard requirement that pulls you to TensorRT-LLM or SGLang. Skip it if you're at single-user scale, on Apple Silicon, or running inference on anything other than NVIDIA / AMD datacenter GPUs.

Rating math: 4.8/5 — the half-point lost is for the operational skill ramp (the engine rewards expertise; punishes "we'll figure it out as we go") and for the kernel coverage gap on niche model architectures. We've recommended vLLM as the production default for two consecutive cycles and the recommendation hasn't aged.

Sources

• vLLM GitHub — release notes, kernel coverage matrix, supported architectures.
• vLLM documentation — operator reference for tensor parallelism, prefix caching, chunked prefill.
• PagedAttention paper (SOSP 2023) — the original architectural argument.

Related

• SGLang — closest credible alternative engine
• TensorRT-LLM — when you've committed to NVIDIA and need the last microsecond
• Ray Serve — the orchestration layer above vLLM in production
• llama.cpp, Ollama — the local-first / single-user side of inference
• /maps/local-ai-agents-2026 — where vLLM sits in the runtime zone
• /authors/fred-oline — about the author