AI glossary

A Large Language Model is a neural network with billions of parameters trained on massive text corpora to predict the ne

Quantization is the process of reducing a model's numeric precision to shrink its memory footprint with minimal quality

Inference is the act of running a trained model to generate predictions, as opposed to training which produces the model

RAG is the pattern of retrieving relevant documents from a knowledge base and including them in the LLM's prompt so the

Hallucination is when an LLM generates plausible-sounding but factually incorrect information — citing papers that don't

Prompt engineering is the practice of crafting model inputs to elicit better outputs without changing the model itself.

LoRA is a parameter-efficient fine-tuning technique that adapts a large pre-trained model by training small low-rank mat

Fine-tuning is continued training of a pre-trained model on a smaller, task-specific dataset. Pre-training builds genera

An embedding is a fixed-length vector representation of text, image, or other input — typically 384-3072 dimensions — wh

Chain-of-thought prompting is asking a model to show its reasoning step-by-step before giving the final answer. It drama

Latency measures how fast you get a response. Two metrics matter for local LLMs: Time to First Token TTFT — wall-clock

GGUF GGML Unified Format is the file format used by llama.cpp and its ecosystem Ollama, KoboldCPP, LM Studio. A single f

Throughput measures how much work a system completes per unit time — typically tokens-per-second across all concurrent r

QLoRA combines LoRA/glossary/lora fine-tuning with 4-bit quantization of the base model. Introduced by Tim Dettmers in 2

Speculative decoding speeds up LLM inference by using a small fast "draft" model to propose the next several tokens, the

Dense retrieval finds documents by computing cosine similarity or dot product between learned vector embeddings of the q

A reranker is a cross-encoder model that scores query/document pairs jointly concatenated as input, producing a relevanc

Hybrid retrieval combines dense and sparse retrieval, typically by union-then-rerank or reciprocal rank fusion RRF. The

BM25 is the canonical sparse-retrieval algorithm: a TF-IDF variant that saturates term frequency a token appearing 100 t

Sparse retrieval scores documents by lexical overlap with the query — high-dimensional vectors where most entries are ze

The KV cache stores the key and value tensors from previous attention computations so the model doesn't recompute them a

The context window is the maximum number of tokens a model can attend to at once — both prompt and previously generated

A token is the smallest unit of text a language model processes. Most modern models use subword tokenization, where comm

Self-attention computes a weighted representation of every position in a sequence by comparing each token against every

Tokenization is the process of converting text into the numeric tokens a model can process. Modern systems use subword t

Prefill is the first phase of LLM inference: the model processes the entire prompt in a single parallel pass, building u

Decode is the second phase of LLM inference: generating one output token at a time, autoregressively. Each decode step d

Flash Attention is a memory-efficient implementation of the attention mechanism that reduces memory usage from On² to On

PagedAttention is the memory layout introduced by vLLM that stores the KV cache in fixed-size blocks pages, like virtual

Sampling is the process of converting model logits into output tokens. Common strategies: greedy temperature 0, random s

Temperature 0 disables sampling entirely — the model picks the highest-logit token at every step. Equivalent to greedy d

Deterministic decoding means same prompt → same output, every time. Achieved by setting temperature to 0 always pick the

A random seed initializes the pseudo-random generator that drives sampling at temperature > 0. Same seed + same prompt +

RMSNorm is a simpler variant of LayerNorm that normalizes activations by their root-mean-square instead of their varianc

YaRN is a context-extension method that modifies RoPE frequencies to let a model trained on, say, 8K context generalize

SwiGLU is a gated feed-forward activation: W1·x ⊙ swishW2·x · W3, replacing the standard MLP's GELU/ReLU in modern trans

ALiBi is a positional encoding scheme that biases attention scores by a linear function of token distance, instead of in

Mirostat is a sampling algorithm that targets a fixed perplexity-like "surprise" level tau instead of a fixed top-p or t

Continuous batching sometimes "iteration-level scheduling" is a serving optimization where new requests join the active

Prefix caching stores the KV cache from previous requests so a new request that shares a prefix system prompt, few-shot

Request batching packs multiple inference requests into a single forward pass to amortize the cost of loading model weig

MPS is Apple's high-level Metal-based compute library, exposed in PyTorch as the mps device backend. Calling model.to"mp

FAISS Facebook AI Similarity Search is a C++/Python library for fast approximate nearest-neighbor search over dense vect

GGML is the C/C++ tensor library that underlies llama.cpp, whisper.cpp, and the original GGUF format. It provides quanti

Vulkan compute is the cross-vendor GPU compute API from Khronos. llama.cpp ships a Vulkan backend that runs on AMD, Inte

DirectML is Microsoft's GPU-agnostic ML acceleration API, layered on DirectX 12. It works on any Windows-supported GPU —

Expert parallelism is a parallelism strategy specific to MoE models: each GPU holds a different subset of the experts, a

Mixture of Experts is a neural network architecture where multiple specialized sub-networks "experts" exist, but only a

Decoder-only is the architecture of GPT, Llama, Qwen, Mistral, DeepSeek, and almost every modern open-weight LLM. The mo

A dense model activates every parameter on every forward pass — the default architecture for transformers like Llama, Qw

MoE routing is the gating mechanism that decides which experts a token activates in a Mixture-of-Experts layer. Top-k ro

Encoder-decoder transformers T5, BART, original "Attention is All You Need" architecture have two halves: an encoder rea

VRAM is the dedicated memory on a GPU. For local AI, VRAM capacity is the single most important spec — it determines whi

CUDA Compute Unified Device Architecture is NVIDIA's parallel-computing platform and the dominant API for GPU-accelerate

MLX is Apple's open-source array framework optimized for Apple Silicon. The Apple equivalent of PyTorch + CUDA, with fir

Tensor parallelism splits each transformer layer's weight matrices across multiple GPUs. Card 0 holds the first half of

ROCm Radeon Open Compute is AMD's open-source equivalent of NVIDIA's CUDA. It's required for any meaningful AMD GPU infe

GDDR7 uses PAM3 signaling to push per-pin rates to 28–32 Gbps in first-gen products 2025, with a path to 40+ Gbps. RTX 5

Pipeline parallelism a.k.a. "layer split" in llama.cpp parlance puts whole layers on different GPUs. Card 0 handles laye

Unified memory is a memory architecture where CPU and GPU share the same physical RAM pool, eliminating CPU↔GPU copies.

NVLink is NVIDIA's proprietary GPU-to-GPU interconnect, used to bind multiple data-center GPUs into a coherent memory fa

NVSwitch is the crossbar that connects 8 or in NVL72, 72 GPUs into a single all-to-all NVLink fabric. Each GPU talks to

Q4KM is the most-downloaded GGUF quantization on Hugging Face — the default tradeoff for local inference. It mixes 6-bit

AWQ Activation-aware Weight Quantization is a 4-bit quantization method designed for fast inference on NVIDIA GPUs. It's

Q5KM is a mixed-precision GGUF quantization averaging ~5.7 bits per parameter. Attention and feed-forward weights use 6-

Q80 is llama.cpp's simplest 8-bit GGUF quantization: weights in INT8, one FP16 scale per 32-element block, no zero-point

GPTQ Generative Pre-trained Transformer Quantization is a one-shot post-training quantization method that uses approxima

Q40 is the original llama.cpp 4-bit quantization: INT4 weights with one FP16 scale per 32-element block, no zero-point,

EXL2 is the ExLlamaV2 quantization format. NVIDIA-only, single-stream-throughput-optimized. Allows fractional bit-rates

Q3KM is a 3-bit GGUF K-quant averaging ~3.9 bits per parameter. It's the smallest format that still produces usable outp

Q2K is 2-bit GGUF quantization averaging ~3.0 bits per parameter with mandatory 4-bit scales and importance metadata. It

An AI agent is software that uses an LLM to decide what to do, takes actions, observes results, and iterates toward a go

Function calling also called tool use is a capability where the model emits structured JSON requesting that specific too

Tool calling also called function calling is a model's structured output capability where it produces JSON-shaped tool i

MCP is an open protocol introduced by Anthropic in late 2024 for connecting AI agents to tools and data sources in a sta

A Reactive Agent selects actions based solely on its current percepts and a fixed set of condition-action rules, without

Tokens per second tok/s is the most-cited LLM throughput metric, but it's also the most-misunderstood. It splits into tw

TTFT time-to-first-token is the latency between sending a prompt and receiving the first generated token. It's dominated

Throughput is aggregate tokens generated per second across all in-flight requests; latency is wall-clock time for a sing

GSM8K is a benchmark of 8,500 grade-school math word problems requiring 2–8 reasoning steps. Models are scored by whethe

pass@1 is the probability that a model's first generated solution passes the unit tests for a coding problem, computed f

Differential Privacy is a mathematical framework that provides a formal guarantee that the output of an analysis reveals

A/B Testing in ML compares two model variants — a control current production model and a treatment candidate model — by

Model Monitoring continuously tracks the health and performance of deployed ML models by measuring: 1 prediction quality

Model Versioning tracks the evolution of ML models over time by assigning unique identifiers to each trained artifact an

A Model Registry is a centralized catalog that stores and versions trained models along with their metadata — training d

Shadow Deployment also called dark launch or shadow mode runs a candidate model in production alongside the current mode