Generative AI (GenAI)

Generative AI (GenAI) refers to machine learning models that produce new content—text, images, audio, code, or video—by learning patterns from training data. Unlike discriminative models that classify or predict labels, GenAI models generate outputs that resemble the training distribution. Operators encounter GenAI when running local models like Llama (text), Stable Diffusion (images), or Bark (audio). The key practical constraint is VRAM: larger models generate higher-quality outputs but require more memory. For example, a 7B-parameter text model at 4-bit quantization fits in ~5 GB VRAM, while a 70B model needs ~40 GB, limiting local deployment on consumer GPUs.

GenAI models are typically based on architectures like transformers (for text) or diffusion models (for images). They are trained on vast datasets to predict the next token (text) or denoise an image (diffusion). At inference, the model generates outputs autoregressively (text) or iteratively (images). Operators interact with GenAI through runtimes like llama.cpp, Ollama, or vLLM for text, and ComfyUI or Automatic1111 for images. Key operator concerns: quantization reduces model size and VRAM usage at a quality cost; context length limits how much text can be generated coherently; and token generation speed (tok/s) depends on hardware, model size, and quantization. GenAI models also require careful prompt engineering to steer output quality.

An operator with an RTX 3090 (24 GB VRAM) can run Llama 3.1 8B at Q4_K_M (5 GB) with 8K context, achieving ~40 tok/s. The same card can run Llama 3.1 70B at Q4_K_M (40 GB) only with system-RAM offload, dropping to ~3 tok/s. For image generation, Stable Diffusion XL requires ~8 GB VRAM for a single image at 1024x1024, taking ~10 seconds on the same card.

When an operator runs ollama run llama3.1:8b, the runtime loads the quantized model into VRAM and generates text token by token. The operator sees a prompt input, then the model streams output. If VRAM is insufficient, Ollama falls back to CPU offload, and tokens/sec drops noticeably. In LM Studio, the operator selects a model from the library, adjusts context length and offloading settings, then chats or runs a script. For image GenAI, the operator loads a Stable Diffusion checkpoint in ComfyUI, sets the prompt and resolution, and clicks 'Queue Prompt' to generate.