Dropout

Dropout is a regularization technique used during neural network training where randomly selected neurons are ignored (dropped out) with a probability p per forward pass. This prevents co-adaptation of neurons, forcing the network to learn more robust features. During inference, dropout is disabled, and weights are scaled by (1-p) to maintain expected output magnitude. Operators encounter dropout when fine-tuning models: it's a hyperparameter (e.g., dropout=0.1) in config files or training scripts. Higher dropout increases regularization but may slow convergence.

Dropout was introduced by Srivastava et al. (2014) as a way to reduce overfitting in deep neural networks. During each training iteration, each neuron (except those in the output layer) has an independent probability p of being temporarily removed from the network. This means the network trains as an ensemble of many thinned sub-networks. The dropout rate p is typically set between 0.2 and 0.5 for hidden layers. For recurrent networks, variational dropout (same mask per time step) is common. During inference, dropout is turned off, and weights are multiplied by (1-p) to compensate for the increased number of active neurons. In practice, dropout is often used in transformer fine-tuning (e.g., attention dropout, feed-forward dropout) and can be critical for small datasets. Operators should note that dropout increases training time slightly but can significantly improve generalization.

When fine-tuning a Llama 3.2 1B model on a custom dataset using Hugging Face Transformers, you might set dropout=0.1 in the training arguments. This means 10% of the neurons in the attention and feed-forward layers are randomly dropped each forward pass during training. On a single RTX 4090, this adds negligible overhead (~1-2% slower per epoch) but can reduce overfitting if the dataset has only a few thousand examples.

In a fine-tuning script using Hugging Face Transformers, dropout is configured via the model config: config.attention_dropout = 0.1 and config.hidden_dropout_prob = 0.1. When using LoRA (common in local AI), dropout is applied to the LoRA layers via lora_dropout=0.05 in the PeftConfig. During inference, dropout is automatically disabled—no operator action needed. If you see poor generalization, increasing dropout in the training config is a typical adjustment.