Backpropagation

Backpropagation is the algorithm used to train neural networks by computing gradients of the loss function with respect to each weight. It works by propagating the error backward from the output layer to the input layer, applying the chain rule of calculus. In practice, operators encounter backpropagation only during fine-tuning or training, not during inference. The computational cost of backpropagation is roughly double that of a forward pass, and it requires storing intermediate activations for gradient computation, which increases VRAM usage significantly. For example, fine-tuning a 7B model with LoRA still requires backpropagation through the adapter layers, consuming more memory than inference alone.

Backpropagation consists of two phases: a forward pass where inputs are fed through the network to compute predictions and loss, and a backward pass where gradients are calculated from the loss back to each weight. The gradients are then used by an optimizer (e.g., AdamW) to update weights. The key memory cost comes from storing activations at each layer during the forward pass, as they are needed for gradient computation in the backward pass. This means training a model requires roughly 2-4x the VRAM of inference for the same model size. Techniques like gradient checkpointing trade compute for memory by recomputing activations during the backward pass. Backpropagation is essential for fine-tuning, but operators running local inference never use it unless they are training or fine-tuning models.

Fine-tuning Llama 3.1 8B with LoRA on a single RTX 4090 (24 GB VRAM) requires backpropagation. The forward pass computes loss on a batch of text, then backpropagation computes gradients for the LoRA adapter weights. The VRAM usage jumps from ~16 GB for inference to ~22 GB for training with a batch size of 1. If VRAM is insufficient, the operator can reduce batch size or enable gradient checkpointing, which trades slower training for lower memory.

When using Hugging Face Transformers with Trainer, backpropagation happens automatically. The operator calls trainer.train(), which loops over batches: forward pass, loss computation, loss.backward() (backpropagation), and optimizer step. In Unsloth or Axolotl, the same process occurs under the hood. Operators can monitor VRAM usage with nvidia-smi; a spike during backpropagation is normal. For inference-only workflows (e.g., ollama run), backpropagation never runs.