Tensor Parallelism

Tensor parallelism splits each transformer layer's weight matrices across multiple GPUs. Card 0 holds the first half of every weight tensor; card 1 holds the second half. On every forward pass, both cards compute their share, then perform an all-reduce to combine results. The all-reduce traffic is roughly proportional to model hidden size × batch size × tokens.

This is what vllm serve --tensor-parallel-size N and sglang.launch_server --tp N enable. The operator-critical insight: tensor parallelism is latency-friendly when interconnect is fast (NVLink, NVLink-Switch) and painful over slow interconnect (PCIe-only, Ethernet). On dual RTX 3090 with NVLink: extracts close to theoretical throughput. On dual RTX 4090 (no NVLink, PCIe-only): 10-20% slower than NVLink-equivalent due to all-reduce going over the slower bus.

When to use TP: same-GPU symmetric multi-card setups with fast interconnect; production serving where multi-tenant throughput matters. When NOT to use TP: asymmetric GPUs (use pipeline-parallel/layer-split via llama.cpp instead), latency-critical single-stream workloads where 2× TP-2 replicas may beat 1× TP-4 (counter-intuitive but real on PCIe), or single-node deployments where the model fits on one card.

An operator with dual RTX 3090s connected via NVLink deploys a 70B model with vllm serve --tensor-parallel-size 2, splitting every weight matrix in half across both cards. Because NVLink provides high-bandwidth direct GPU-to-GPU transfer, the per-layer all-reduce overhead stays small and the setup approaches the throughput of a single card with double the VRAM. A colleague replicates the setup on dual RTX 4090s without NVLink — same tensor-parallel config, but the all-reduce now crosses PCIe, and single-stream throughput comes in noticeably behind the NVLink pair despite the 4090s having more raw compute. For that PCIe-only rig, switching to pipeline-parallelism (layer split) instead of tensor parallelism reduces cross-card traffic from every layer to once per boundary, recovering most of the lost throughput at the cost of some latency under concurrent load.