Artificial Intelligence (AI)

Artificial Intelligence (AI) refers to systems that perform tasks typically requiring human intelligence, such as reasoning, learning, and decision-making. In local AI, operators run AI models directly on their own hardware—like an RTX 4090 or Apple M-series—rather than relying on cloud APIs. This means models handle inference locally, processing inputs (e.g., text prompts) and generating outputs (e.g., completions) using the rig's VRAM and compute. The term is broad, covering everything from small language models (e.g., Llama 3.2 1B) to large ones (e.g., Llama 3.1 70B), but in practice, operators encounter AI as a runtime that loads weights, runs inference, and manages resource constraints.

AI as a field dates to the 1950s, but the local AI operator cares about the modern incarnation: deep neural networks, particularly transformer-based language models. These models are trained on vast datasets to predict next tokens, and then quantized and distributed as weight files (e.g., GGUF, safetensors). Running AI locally means loading these weights into memory—ideally VRAM—and executing forward passes to generate text, images, or audio. The operator's experience is shaped by hardware limits: a 7B parameter model at Q4_K_M quant requires ~4.5 GB VRAM, fitting on most consumer GPUs, while a 70B model needs ~40 GB, often exceeding single-GPU capacity. Runtimes like llama.cpp and Ollama abstract the complexity, but operators still manage trade-offs between model size, quantization level, context length, and token generation speed. AI is not a monolithic entity; it's a stack of model architecture, weights, runtime, and hardware—each layer affecting what's possible on a given rig.

An operator with an RTX 3090 (24 GB VRAM) can run Llama 3.1 8B at Q4_K_M (5 GB) with a 32K context window, achieving ~40 tok/s. The same operator trying Llama 3.1 70B at Q4_K_M (40 GB) would need to offload layers to system RAM, dropping speed to ~3 tok/s. This illustrates how AI model choice is constrained by VRAM and quantization, not just model capability.

When using Ollama, an operator runs ollama run llama3.1:8b to start an AI model. The runtime downloads the GGUF file, loads it into VRAM, and processes prompts interactively. If VRAM is insufficient, Ollama falls back to CPU offload—visible in the output as slower token generation. Operators monitor this with ollama ps to see memory usage and adjust by choosing smaller models or lower quantizations.