Hyperparameter

A hyperparameter is a configuration variable set before training begins that controls the learning process, not a parameter learned from data. In local AI, operators encounter hyperparameters when fine-tuning models: learning rate, batch size, number of epochs, and LoRA rank. These values directly affect training speed, model quality, and VRAM usage. For example, a batch size too large can exceed GPU memory, while a learning rate too high may cause divergence. Hyperparameters are tuned empirically—operators run multiple training runs with different values to find what works for their dataset and hardware.

Hyperparameters fall into two categories: model hyperparameters (e.g., number of layers, hidden size) and training hyperparameters (e.g., learning rate, batch size, optimizer choice). Model hyperparameters define the architecture and are fixed before training; changing them means using a different model. Training hyperparameters control the optimization loop. Key ones include: learning rate (step size for gradient updates), batch size (samples per forward/backward pass), epochs (full passes over the dataset), and weight decay (regularization). For LoRA fine-tuning, rank and alpha are additional hyperparameters that control the expressiveness of the adapter. Operators often use learning rate schedulers (e.g., cosine decay) to adjust the rate during training. Hyperparameter search methods like grid search or Bayesian optimization help find good values, but are computationally expensive on consumer hardware. A common heuristic: start with values from similar published fine-tunes, then adjust based on loss curves and VRAM usage.

Fine-tuning Llama 3.1 8B with LoRA on an RTX 4090 (24 GB VRAM). A typical hyperparameter set: learning rate 2e-4, batch size 4 (due to VRAM limit), LoRA rank 16, alpha 32, 3 epochs. If batch size is set to 8, the runtime will throw an out-of-memory error. If learning rate is set to 1e-2, loss may spike and training fails to converge. Operators monitor loss and adjust these values between runs.

In Hugging Face Transformers, hyperparameters are passed to TrainingArguments: TrainingArguments(output_dir='./results', learning_rate=2e-4, per_device_train_batch_size=4, num_train_epochs=3). In llama.cpp's fine-tuning script, they are set via command-line flags: --learning-rate 2e-4 --batch-size 4 --epochs 3. Operators often log these in a config file or notebook cell, then iterate by changing one value at a time and comparing loss curves.