Bias-Variance Tradeoff

The bias-variance tradeoff describes the tension between a model's ability to fit training data closely (low bias) and its sensitivity to variations in the training data (high variance). In practice, high-bias models underfit—they miss patterns—while high-variance models overfit—they memorize noise instead of generalizing. Operators encounter this when choosing model size: a 7B parameter model has higher bias (simpler patterns) than a 70B model, which has lower bias but higher variance (needs more data and regularization to avoid overfitting). The tradeoff guides decisions on training duration, dataset size, and regularization techniques like dropout or weight decay.

Fine-tuning a small 1.5B model on a custom dataset of 500 examples: if training runs for too many epochs, the model may memorize the 500 examples (high variance, poor generalization). If training stops early, it may not capture the patterns (high bias). Operators balance this by monitoring validation loss—when it starts rising while training loss drops, variance is winning. Using dropout (e.g., 0.1) or weight decay (e.g., 0.01) can reduce variance without increasing bias too much.

In Hugging Face Transformers, when training a model with Trainer, operators set evaluation_strategy="steps" and load_best_model_at_end=True to monitor validation loss. If the model overfits, they increase weight_decay (e.g., from 0.01 to 0.1) or add dropout via config.dropout. In llama.cpp's training mode (e.g., llama-train), operators adjust --lora-r (rank) and --lora-alpha to control variance—lower rank reduces variance but may increase bias.