Pipeline Parallelism

Pipeline parallelism (a.k.a. "layer split" in llama.cpp parlance) puts whole layers on different GPUs. Card 0 handles layers 0-39; card 1 handles layers 40-79. On every forward pass, the activation tensor crosses the bus once at the layer-boundary transition, not on every layer like tensor parallelism.

The trade-off: pipeline parallelism is bandwidth-friendlier than tensor parallelism on slow interconnect (PCIe-only multi-GPU, Thunderbolt cluster, multi-machine over Ethernet) because cross-card traffic is once per token instead of every layer. The downside: inherently sequential — card 1 sits idle while card 0 is computing the first half of layers, so single-stream throughput is limited to the per-card throughput. You only win latency parallelism via concurrent throughput.

Pipeline parallelism is the right answer for: asymmetric GPU pairs (mixed RTX 4090 + RTX 3090 — the ratio handles the throughput difference), PCIe-only multi-GPU (no NVLink penalty matters less), multi-machine clusters (Exo, Petals, Hyperspace pods). vLLM supports it via --pipeline-parallel-size; llama.cpp via --tensor-split. Often combined with TP in hybrid configurations on large datacenter clusters.

An operator with a mismatched pair — an RTX 4090 (24GB) and an older RTX 3090 (24GB) sitting in the same box on PCIe only, no NVLink — can't use tensor parallelism efficiently because tensor-parallelism assumes symmetric per-card throughput and the slower 3090 would bottleneck every layer's all-reduce. Instead they run llama.cpp with --tensor-split to assign more layers to the 4090 and fewer to the 3090, roughly balancing wall-clock time per card despite the compute asymmetry. Cross-card traffic drops to once per layer-boundary transition instead of every layer, which matters more on PCIe than it would with NVLink. The trade-off: single-stream latency is bounded by the sequential hand-off between cards, so a second concurrent request doesn't parallelize for free — throughput scaling requires running multiple pipeline replicas, not just one deeper pipeline.