Model Serving
Definition pending
We've cataloged "Model Serving" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
Model serving is the runtime that executes model inference — loading the model into memory and processing requests. For LLMs, the serving infrastructure (vLLM, TGI, Ollama) determines throughput, latency, and concurrent-user capacity. Serving choice impacts operational costs more than model choice.
Workflow example
Model serving selection: (1) single-user development: Ollama (llama.cpp), simplest setup, (2) multi-user SaaS: vLLM (PagedAttention, continuous batching, 10–50× throughput of llama.cpp for concurrent users), (3) enterprise: NVIDIA Triton Inference Server (multi-model, multi-framework), (4) CPU-only: llama.cpp with CPU backend, ONNX Runtime, (5) benchmark YOUR workload: different serving engines excel at different batch sizes, sequence lengths, and hardware — test before deciding.