Tensor Parallelism

Tensor parallelism splits each transformer layer's weight matrices across multiple GPUs. Card 0 holds the first half of every weight tensor; card 1 holds the second half. On every forward pass, both cards compute their share, then perform an all-reduce to combine results. The all-reduce traffic is roughly proportional to model hidden size × batch size × tokens.

This is what vllm serve --tensor-parallel-size N and sglang.launch_server --tp N enable. The operator-critical insight: tensor parallelism is latency-friendly when interconnect is fast (NVLink, NVLink-Switch) and painful over slow interconnect (PCIe-only, Ethernet). On dual RTX 3090 with NVLink: extracts close to theoretical throughput. On dual RTX 4090 (no NVLink, PCIe-only): 10-20% slower than NVLink-equivalent due to all-reduce going over the slower bus.

When to use TP: same-GPU symmetric multi-card setups with fast interconnect; production serving where multi-tenant throughput matters. When NOT to use TP: asymmetric GPUs (use pipeline-parallel/layer-split via llama.cpp instead), latency-critical single-stream workloads where 2× TP-2 replicas may beat 1× TP-4 (counter-intuitive but real on PCIe), or single-node deployments where the model fits on one card.