Ray Serve

Distributed model serving on top of Ray. Lets you stitch vLLM / SGLang / custom runtimes into a multi-replica, multi-model deployment with autoscaling, traffic splitting, and pipeline composition. The orchestration layer above raw inference engines.

By Fredoline Eruo·Last verified Jun 12, 2026·33,000 GitHub stars

Overview

Stack & relationships

How Ray Serve relates to other entries in the catalog — recommended pairings, alternatives, dependencies, and edges to avoid. Each edge carries a one-line operator note from our editorial team.

Ray Serve ↔ ecosystem

Recommended stack

Commonly deployed with
vLLM
Ray Serve in front of vLLM is the canonical K8s production pattern — autoscaling replicas, traffic splitting, canary deploys.
Commonly deployed with
vLLM
Ray Serve orchestrates vLLM replicas in K8s. The canonical 'we replaced our OpenAI bill' production stack.
Commonly deployed with
SGLang
Same canonical pattern as vLLM — Ray Serve in front for K8s-grade autoscaling. SGLang's cross-replica RadixAttention sync compounds the cluster-level wins.
Commonly deployed with
SGLang
Same orchestration layer above SGLang as above vLLM. Ray Serve doesn't care which engine is underneath — that's the architectural point.
Commonly deployed with
vLLM
Ray Serve in front of vLLM = the canonical K8s production pattern. Autoscaling replicas, traffic splitting, canary deploys.
Commonly deployed with
SGLang
Same pattern as Ray Serve + vLLM. SGLang's cross-replica RadixAttention sync makes the cluster-level wins compound at multi-node scale.

Alternatives

Alternative to
Exo
Exo for Apple-Silicon LAN clusters; Ray Serve for datacenter multi-node. Different hardware targets; non-overlapping operating points.

Featured in these stacks

The L3 execution stacks that pick this tool as a recommended component, with the one-line note explaining the role it plays in each.

Stack · L3·Production tier·Role: Cluster orchestrator (head node + worker placement)
Build a distributed inference homelab stack (May 2026)
Ray Serve is the canonical orchestration layer above vLLM in distributed deployments. Handles worker placement, autoscaling, traffic splitting, canary deploys. Same Ray cluster scales to add SGLang or other engines later — pick the orchestrator first; pick the engine inside it.
Stack · L3·Homelab tier·Role: Orchestration layer for 2×TP-2 replica pattern
Quad RTX 3090 workstation stack — the prosumer 100B-class ceiling
Ray Serve over quad-3090: split into 2 replicas of tensor-parallel-2 each. Higher single-stream throughput than 1×TP-4; better aggregate concurrency. Use when serving 8+ users.

Pros

Composes multiple runtimes (vLLM + custom) under one routing layer
Autoscaling + canary deploys built in
Same stack handles training + serving

Cons

Ray adds operational surface area
Overkill for single-model deployments

Compatibility

Operating systems	Linux macOS Kubernetes
GPU backends	NVIDIA CUDA AMD ROCm
License	Open source · free (OSS, Apache 2.0) + Anyscale managed

Runtime health

Operator-grade signals on how actively Ray Serve is being maintained, how fresh its measurements are, and what failure classes operators have flagged. Every label below is anchored to a real date or count — we never infer maintainer activity we can't show.