06. Pipeline Parallelism

Pipeline parallelism distributes model layers across GPUs, enabling massive models where tensor parallelism would require excessive collective communication. GPUs process different model stages in sequence, with micro-batches flowing through the pipeline.

The technique partitions the model vertically: earlier layers execute on one set of GPUs, later layers on another. Input tokens flow forward through each stage, then backward for training. The pipeline creates bubbles—periods where GPUs idle waiting for other stages—during the initial and final micro-batches of each batch.

Pipeline scheduling strategies minimize bubbles. GPipe and PipeDream-1F1B represent fundamental tradeoffs: GPipe prioritizes memory efficiency with larger activation recomputation, while 1F1B prioritizes training throughput by overlapping forward and backward passes.

Configuration challenges define pipeline parallelism practicality. The number of pipeline stages determines per-GPU memory reduction but also pipeline depth. Batch size must divide evenly across pipeline stages to avoid micro-batch remainder complications. Learning rate scaling may require adjustment for larger global batch sizes.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.