Ray Serve

What it is and how it works

Ray Serve is the model-serving library built on top of Ray, the distributed compute framework originally developed at UC Berkeley's RISELab and now maintained by Anyscale. Where most inference servers (vLLM, TGI, llama.cpp server) are designed to serve one model well, Ray Serve sits a layer above that: it's a framework for composing multiple models, runtimes, and business logic into a single deployment graph, and for scaling that graph across a cluster of machines. It doesn't replace vLLM or SGLang — it wraps them. A common pattern is to run vLLM as the actual inference engine inside a Ray Serve "deployment," and let Ray handle replica placement, autoscaling, request routing, and composition with other deployments (a reranker, a guardrail model, a preprocessing step, a completely different model for a different endpoint).

Architecturally, Ray Serve inherits Ray's core abstractions: actors and tasks distributed across a Ray cluster (a head node plus worker nodes). Each "deployment" in Ray Serve is backed by one or more replicas, which are Ray actors — long-lived Python processes that hold a model in memory (or a handle to a GPU-resident model) and process requests. A lightweight HTTP/gRPC ingress layer (built on Starlette/FastAPI-style routing) sits in front, and Ray's own scheduler decides which physical GPU/CPU each replica lands on. Because it's Python-native and runs inside the Ray runtime, you can write arbitrary orchestration logic — conditional routing, multi-step pipelines, ensemble calls to several models, A/B splits — as plain Python classes decorated with @serve.deployment, and Ray Serve handles the distributed plumbing (queuing, batching, retries, health checks) underneath.

The other defining trait is that Ray Serve shares its runtime with the rest of the Ray ecosystem — Ray Train, Ray Data, Ray Tune. This means the same cluster that trains or fine-tunes a model can serve it, and Ray Data pipelines can feed batch-inference jobs through the same deployment primitives used for online serving. That's a genuinely different design center than dedicated inference servers, which assume the model is already trained and just need to serve it as fast as possible.

Deployment patterns

On a single machine, Ray Serve is arguably more setup than most local-AI operators need — you're running a full Ray head node process (GCS, dashboard, object store) just to serve one model, when vLLM's own OpenAI-compatible server would do the same job with a fraction of the operational surface. That said, it does work fine on a solo workstation for prototyping a multi-model pipeline before scaling it out, and ray start --head plus a Python deployment script is enough to get a local endpoint running with the dashboard for observability.

Where Ray Serve actually earns its complexity is homelab/cluster and team-server scenarios: multiple GPU nodes, multiple models that need independent scaling (a large generation model plus a small embedding model plus a reranker, each with different replica counts and autoscaling targets), or a requirement to route traffic between model versions for canary testing. A typical production shape is a Kubernetes cluster running KubeRay (the Ray Kubernetes operator), with Ray Serve deployments defined declaratively via a RayService custom resource, GPU node pools tainted for inference workloads, and Ray Serve's built-in autoscaler adjusting replica counts based on queue depth. Anyscale's managed offering removes the cluster-ops burden entirely for teams that don't want to run KubeRay themselves, at the cost of being a paid, hosted layer on top of the open-source core.

How it compares

Against vLLM or SGLang directly: those are inference engines, not orchestrators — Ray Serve commonly deploys them as its worker processes rather than competing with them. If you only need to serve one model as fast as possible, vLLM's native server (with its own continuous batching and PagedAttention) is simpler to operate and has less overhead than wrapping it in Ray actors.

Against BentoML: both are Python-native model-serving frameworks with composable deployment graphs, but BentoML is lighter-weight and more focused purely on packaging/serving a model as a service, while Ray Serve inherits the full weight (and full power) of the Ray distributed runtime, including multi-node autoscaling and integration with Ray's training stack.

Against KServe (the Kubernetes-native model serving standard, often paired with Triton or vLLM backends): KServe is more purely declarative/YAML-driven and fits naturally into existing K8s/Istio tooling, whereas Ray Serve favors expressing routing and composition logic in actual Python code, which is more flexible for complex pipelines but less familiar to platform teams used to CRDs and GitOps.

Against a simple reverse-proxy/load-balancer in front of several standalone vLLM instances: that approach is far simpler to reason about for a fixed set of models, but Ray Serve's autoscaling, dynamic replica placement, and in-process pipeline composition solve real problems once you have more than a couple of models or need traffic splitting without hand-rolling it.

Best use cases and honest limitations

Ray Serve is the right tool when you're stitching together multiple models or runtimes behind one routing layer — say, a RAG pipeline with an embedding model, a reranker, and an LLM, each scaled independently — or when you want the same infrastructure to handle both training/fine-tuning and serving. Its autoscaling and canary-deploy primitives are genuinely built-in rather than bolted on, which matters for teams running live traffic against multiple model versions.

It is a poor fit for the common local-AI case of "I want to serve one model on my machine or homelab box." The operational surface area — a Ray cluster, GCS, object store, dashboard, and the mental model of actors/deployments — is real overhead for a single-model deployment that vLLM, llama.cpp, or Ollama would handle with far less to reason about and debug. Teams without existing Ray or Kubernetes experience should expect a nontrivial ramp-up cost. It's best understood as infrastructure for organizations that already have (or are willing to take on) distributed-systems complexity in exchange for composition and scaling flexibility, not as a lightweight local inference tool.