Triton Inference Server

Triton Inference Server is an open-source inference serving software by NVIDIA that manages multiple AI models across GPU and CPU hardware, handling request batching, model versioning, and dynamic model loading. Operators encounter it when deploying models in production environments where they need to serve multiple models (e.g., LLMs, vision models) from a single endpoint, with automatic GPU scheduling and concurrent request handling. It supports frameworks like TensorRT, ONNX, PyTorch, and custom backends, and is commonly used in data centers or edge servers, not on single consumer GPUs.

Triton Inference Server is designed for high-throughput, low-latency inference serving. It decouples model loading from inference execution, allowing models to be loaded/unloaded without restarting the server. Key features include concurrent model execution on the same GPU, dynamic batching (combining multiple requests into a single batch for GPU efficiency), and model pipelines (chaining multiple models). It also supports model ensembles and BLS (Business Logic Scripting) for custom preprocessing/postprocessing. For local AI operators, Triton is relevant when scaling from single-model experiments to multi-model production deployments, but it introduces complexity (containerization, GPU scheduling) that is unnecessary for single-user local inference. It competes with vLLM for LLM serving but is more general-purpose.

An operator running an LLM chatbot and an image classifier on a server with two RTX 4090s could use Triton to serve both models from a single endpoint. Triton would load the LLM on GPU 0 and the classifier on GPU 1, batch incoming requests, and return responses. Without Triton, the operator would need separate processes for each model, manually managing GPU memory and request routing.

To deploy a model with Triton, an operator creates a model repository folder structure with config.pbtxt files specifying model name, backend (e.g., tensorrt_llm), and instance groups. They then run the Triton container: docker run --gpus all -v /path/to/model_repo:/models nvcr.io/nvidia/tritonserver:24.08-trtllm-python-py3. Clients send requests via HTTP/gRPC to the server's endpoint (e.g., http://localhost:8000/v2/models/llm/infer). This workflow replaces direct model loading in Python scripts.