Vanishing Gradient

The vanishing gradient problem occurs when gradients used to update model weights become extremely small as they are backpropagated through many layers. In deep neural networks, gradients shrink exponentially, causing early layers to learn very slowly or not at all. This is especially relevant for recurrent neural networks (RNNs) and deep transformers, where long sequences or many layers amplify the effect. Operators encounter this when training models: if loss stops decreasing after a few epochs, vanishing gradients may be the cause. Modern architectures like transformers use residual connections and layer normalization to mitigate this, but it remains a concern when fine-tuning very deep models or training from scratch.

When fine-tuning a 70B parameter model like Llama 2 70B on a single GPU, the gradient signal must propagate through 80 transformer layers. Without residual connections, the gradient at layer 1 would be near zero, and the model would fail to learn. This is why all modern LLMs use skip connections—they allow gradients to flow directly to early layers.

In a Hugging Face Transformers training script, if you set model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf') and train with a deep custom head, you might see the loss plateau after a few steps. Checking gradient norms via torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) can reveal vanishing gradients if norms are near zero. Using residual connections and proper initialization (e.g., torch.nn.init.xavier_uniform_) helps.