Gradient Descent

Gradient descent is an optimization algorithm that iteratively adjusts model weights to minimize a loss function. In local AI training, it computes the gradient (direction and magnitude of error) for each weight using backpropagation, then updates weights in the opposite direction of the gradient. The step size is controlled by the learning rate. Operators encounter gradient descent when fine-tuning models on their own hardware—each pass over a batch of data computes gradients, and weight updates accumulate across batches. The process repeats until the loss converges, often requiring significant VRAM for storing intermediate gradients and optimizer states (e.g., Adam uses ~8 bytes per parameter).

Gradient descent comes in three main variants relevant to local training: batch, stochastic (SGD), and mini-batch. Batch gradient descent computes gradients over the entire dataset, which is infeasible for local hardware due to VRAM limits—a single pass over a 7B model with 4K context can exceed 24 GB. SGD updates weights after each sample, introducing noise that can help escape local minima but requires careful learning rate scheduling. Mini-batch gradient descent (the standard) uses batches of 1–64 samples, balancing stability and memory. The optimizer choice (SGD, Adam, AdamW) affects memory: Adam stores first and second moment estimates, doubling the per-parameter memory. For local operators, gradient accumulation (summing gradients over multiple batches before updating) simulates larger batch sizes without exceeding VRAM. Learning rate schedules (cosine, linear, warmup) are critical—too high causes divergence, too low stalls training. Modern fine-tuning methods like LoRA reduce gradient memory by training low-rank adapters instead of full weights.

When fine-tuning Llama 3.1 8B with LoRA on an RTX 4090 (24 GB VRAM), gradient descent updates only the LoRA adapter weights (~0.1% of parameters). With a batch size of 4 and sequence length 2048, the runtime stores gradients for ~8M parameters, consuming ~64 MB (8 bytes per parameter for Adam). Full fine-tuning the same model would require ~64 GB for gradients alone—impossible on consumer hardware. Operators using Unsloth or Axolotl see gradient descent progress via loss curves; a typical run on a single GPU achieves ~1–2 steps per second at batch size 4.

In a typical fine-tuning workflow with Hugging Face Transformers, gradient descent runs inside the Trainer class. The operator sets per_device_train_batch_size=2, gradient_accumulation_steps=4, and learning_rate=2e-4. Each step: forward pass computes loss, backward pass computes gradients via loss.backward(), optimizer step updates weights via optimizer.step(), then optimizer.zero_grad() clears gradients. VRAM usage spikes during backward pass—monitor with nvidia-smi or torch.cuda.memory_summary(). If VRAM exceeds capacity, reduce batch size or enable gradient checkpointing (trades compute for memory).