Neural Networks

Neural networks are the computational architecture behind modern AI models. They consist of layers of interconnected nodes (neurons) that transform input data through weighted connections and nonlinear activation functions. During training, these weights are adjusted to minimize prediction error. For local AI operators, neural networks define the model file you download (e.g., a 7B-parameter Llama model) and determine VRAM requirements: each parameter is stored as a weight, so a 7B model at 16-bit precision needs ~14 GB of VRAM just for weights. Quantization reduces this by lowering precision (e.g., 4-bit ~3.5 GB), trading accuracy for fit on consumer GPUs.

A neural network is composed of an input layer, one or more hidden layers, and an output layer. Each neuron computes a weighted sum of its inputs plus a bias, then applies an activation function (e.g., ReLU, sigmoid) to introduce nonlinearity. Deep neural networks (many hidden layers) can learn hierarchical features. The specific architecture—such as transformer (attention-based) or convolutional (spatial pattern detection)—determines how data flows. For operators, the network's depth and width directly affect inference speed and memory. For example, a 70B-parameter transformer has ~80 layers and 8K hidden dimension, requiring ~140 GB at 16-bit. Quantization to 4-bit compresses weights to ~40 GB, enabling it to run on a single 48 GB GPU (e.g., RTX 6000 Ada) but not on a 24 GB card without offloading.

Consider Llama 3.1 8B: it has ~8 billion parameters, each stored as a weight. At 16-bit float (2 bytes per parameter), the model weighs ~16 GB. Running it on an RTX 4090 (24 GB VRAM) fits with room for context. Quantize to 4-bit (0.5 bytes per parameter) and it drops to ~4 GB, fitting on an RTX 3060 (12 GB) with headroom for a 32K token context. The network's architecture (transformer) also dictates that attention layers scale quadratically with context length, so longer prompts consume more VRAM.

When you run ollama run llama3.1:8b, Ollama loads the neural network weights into VRAM. The runtime first checks available VRAM; if insufficient, it offloads layers to system RAM, slowing tokens/sec from ~40 to ~5. In LM Studio, you can select a quantization level (e.g., Q4_K_M) which directly controls the precision of the neural network's weights—lower precision means smaller VRAM footprint but potential quality loss. In vLLM, the --max-model-len flag limits context length, reducing the memory used by attention layers in the network.