08. Multi-Node Inference

Multi-node inference extends serving beyond single-machine GPU capacity, enabling production deployment of the largest models. The coordination overhead and latency characteristics differ substantially from single-node serving.

Ray Serve and vLLM with pipeline parallelism support multi-node serving natively. Ray Serve distributes actors across the cluster, enabling model replicas to span nodes. vLLM's pipeline parallelism distributes layers across nodes while maintaining tensor parallelism within each node.

Deployment configuration example for Ray Serve:

import ray
from ray import serve
from vllm import LLM

ray.init(address="auto")

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_gpus": 2}
)
class MultiNodeModel:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Llama-2-70b-hf",
            tensor_parallel_size=2,
            pipeline_parallel_size=2
        )
    
    def generate(self, prompt):
        return self.llm.generate(prompt)

serve.run(MultiNodeModel.bind())

Common pitfalls include coordinator bottlenecks where a single node gates requests before distribution. This manifests as latency spikes without corresponding increased GPU utilization. Load balancing configuration must distribute requests to all serving nodes evenly.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.