vLLM

What vLLM actually is

vLLM is the production-default open-source LLM serving engine in 2026. Born out of UC Berkeley's Sky Computing lab and now maintained by Anyscale + a wide industry contributor base, it is the runtime almost every team reaches for once they've outgrown Ollama or llama.cpp and need to actually serve concurrent users without their throughput collapsing.

The single technical idea that put vLLM on the map is PagedAttention — managing the KV-cache in fixed-size blocks the way an OS pages physical memory. That one insight, plus continuous batching, plus competent tensor-parallel implementations, plus a near-OpenAI-shaped HTTP API, is what differentiates a "real" inference server from a single-stream toy.

If you are running a local inference service that more than one person hits — a team chat backend, an internal coding-assistant gateway, a RAG API — vLLM is almost certainly the right answer.

Where it fits in the stack

vLLM lives at the server layer of any serious local-AI deployment:

• Frontend / clients: Open WebUI, LibreChat, AnythingLLM, Continue.dev, or any OAI-compatible app
• Server: vLLM exposes http://host:8000/v1 with a near-OAI-shape API
• Engine: vLLM's own CUDA kernels (PagedAttention, FlashAttention 2/3 integration, MoE kernels)
• Hardware: NVIDIA CUDA primary, AMD ROCm partial, Intel Gaudi via plugin, others experimental

It is not the right layer for a single-user laptop with intermittent inference (use Ollama) and not the right layer when single-stream latency dominates throughput (use ExLlamaV2 or TensorRT-LLM on H100). It is the right layer for "multiple users hit the same model concurrently and I need predictable tail latencies."

Best use cases

• Internal team coding-assistant backend. Pair with Qwen 2.5 Coder 32B AWQ-INT4 on a single RTX 4090 or RTX 5090 and serve a 5-15 person dev team comfortably.
• Multi-tenant RAG gateway. Continuous batching turns concurrent RAG requests with overlapping prefixes into roughly free throughput.
• Cost-controlled OpenAI replacement. Drop vLLM behind an existing OAI-compatible client, point the base URL at it, save the per-token spend.
• Production-grade dual-GPU serving. Tensor-parallel across 2× RTX 3090 or 2× 4090 for 70B-class models with usable concurrent throughput.
• Datacenter rollouts. H100 SXM clusters with NVLink — vLLM is the most common engine on the rented-GPU market.

OS support

If you are on Windows and want vLLM-class throughput, your real options are WSL2 (acceptable) or a dedicated Linux box (recommended).

Hardware / runtime support

• NVIDIA CUDA — first-class. Ampere (3090, A100), Ada (4090, L40S), Hopper (H100, H200), Blackwell (5090, B200) all work.
• AMD ROCm — supported on MI300X for production, RDNA3 consumer cards (RX 7900 XTX) are workable but rougher.
• Intel Gaudi — supported via the vllm-hpu plugin path; the canonical Gaudi inference engine.
• Apple Silicon — experimental, not the recommended path. Use MLX-LM.
• Multi-GPU — tensor-parallel and pipeline-parallel both supported. TP works well over PCIe; pipeline-parallel is the right call for slow interconnects.

For the multi-GPU strategy decision tree see /guides/running-local-ai-on-multiple-gpus-2026.

Model / quant format support

Wider than the casual user expects:

• FP16 / BF16 — reference baseline
• AWQ-INT4 — the production sweet spot for 32B-class on consumer cards
• GPTQ-INT4 — supported, slightly slower kernels than AWQ
• FP8 (E4M3 / E5M2) — Hopper / Blackwell only, the canonical datacenter quant
• GGUF — partial; works but slower than the native quants — use llama.cpp for GGUF
• EXL2 — unsupported; that's ExLlamaV2 territory

For the format-vs-runtime cross-product see /systems/quantization-formats.

Setup path

The Docker path is the production default:

docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model casperhansen/llama-3.1-8b-instruct-awq

The pip path for development:

pip install vllm
vllm serve casperhansen/llama-3.1-8b-instruct-awq --port 8000

That gives you an OpenAI-compatible /v1/chat/completions endpoint immediately.

What breaks first

1. VRAM at concurrent load. --gpu-memory-utilization 0.9 is fine on a quiet card but with 16 concurrent users hitting long-context prompts you OOM. Set it to 0.85 and budget for KV-cache.
2. Wrong CUDA wheel for the driver. vLLM ships separate wheels for CUDA 11.8 / 12.1 / 12.4. Mismatched driver vs wheel = silent CPU fallback or load failure.
3. MoE models without the right kernel. New MoE architectures (Qwen3-MoE, DeepSeek-V3 lineage) sometimes need a vLLM minor version bump before they work.
4. Tensor-parallel on heterogeneous GPUs. TP requires identical cards. 1× 3090 + 1× 4090 will not work as a TP-2 pair.
5. AMD ROCm path drift. Consumer RDNA3 + vLLM is workable on Ubuntu 24.04 + ROCm 6.x but real-world reliability lags the CUDA path. See /errors/rocm-device-not-found.

Alternatives by intent

Best pairings

• Open WebUI / LibreChat — drop the OAI-compatible URL in, multi-user chat over a vLLM backend
• AnythingLLM — workspace-RAG frontend pointed at vLLM
• RTX 4090 + AWQ-INT4 32B — the homelab production default
• H100 SXM + FP8 — the rented-GPU production default
• Continue.dev for IDE-side coding — same OAI-shape config

Who should avoid vLLM

• Single-user laptops. Overkill; use Ollama.
• Apple-only fleets. Wrong stack; use MLX-LM.
• GGUF-only workflows. Possible but not what vLLM optimizes; stick with llama.cpp.
• Operators with no Linux experience. Production vLLM is a Linux deployment; if that's a bridge too far, run Ollama on Windows instead.

Related

• Stacks: /stacks/local-coding-agent, /stacks/dual-3090-workstation, /stacks/dual-4090-workstation
• System guides: /systems/quantization-formats, /guides/running-local-ai-on-multiple-gpus-2026
• Hardware: RTX 4090, RTX 3090, H100 SXM
• Errors: /errors/rocm-device-not-found, /errors/wsl2-gpu-not-detected

Install via pip in a Python 3.10+ venv with CUDA 12.4+: pip install vllm. For production, use the Docker image: docker pull vllm/vllm-openai:latest. Start with vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000. If you hit CUDA OOM on the first inference request (not on model load), the --gpu-memory-utilization default of 0.9 is too aggressive — drop to 0.85. Verify with curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello"}]}'. First run downloads the model checkpoint from HuggingFace — budget 5–20 minutes for a 70B model on a decent connection. Time-to-first-useful-response from zero: ~10 minutes with Docker and a cached model. vLLM uses its own OpenAI-compatible REST API at /v1/chat/completions and /v1/completions. The Prometheus metrics endpoint is at /metrics by default.

Best for: multi-tenant LLM serving on NVIDIA datacenter GPUs (H100, A100, H200, B200), production API backends with >1 concurrent request sustained, LoRA-swapping serving of many fine-tuned adapters from one base model, speculative decoding pipelines where 1.5–2.5× throughput matters. Not suited for: single-user local chat (use Ollama), CPU-only or Apple Silicon inference (use llama.cpp or MLX-LM), GGUF-centric model ecosystems without conversion, embedding-only workloads (use Text Embeddings Inference). The migration point from Ollama to vLLM: sustained >1 concurrent request or throughput requirements above ~100 tok/s aggregate.

Use vLLM when your workload has sustained multi-request concurrency — its continuous batching and PagedAttention deliver 5–24× throughput over naive HuggingFace generate(). Switch to SGLang when prefix cache hit rate exceeds ~50% (agent loops, stable system prompts) — RadixAttention wins 15–40% there. Use TensorRT-LLM when locked to a single NVIDIA GPU SKU (H100/H200/B200) and you need every microsecond of single-request latency; it wins 15–30% but requires a 2-hour engine compile for large models. Pick Ollama or LM Studio for single-user desktop use — they are simpler and cover CPU/Apple Silicon. Use llama.cpp when you need CPU inference, Apple Silicon, Vulkan, or the broadest GGUF quantization coverage.

Problem: CUDA OOM on first inference request but model loaded fine. Fix: Lower --gpu-memory-utilization from 0.9 to 0.85. The KV cache needs headroom beyond weight memory. Check actual VRAM usage with nvidia-smi and tune down until stable. Problem: Requests stall at high concurrency. Fix: Enable --enable-chunked-prefill. Long prompts without chunked prefill block the scheduler from interleaving decode steps. Problem: Prefix cache never hits despite identical system prompts. Fix: Any change to the system prompt — including one trailing space — invalidates the cache. Template variable data into user messages instead of modifying the system prompt. Use --enable-prefix-caching explicitly and monitor cache hit rate via Prometheus metrics.