Inference API
Definition pending
We've cataloged "Inference API" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
An inference API is the external interface that serves model predictions — REST or gRPC endpoint that takes input, returns output. For LLMs: the OpenAI-compatible API format has become the standard. For local AI: Ollama and llama.cpp expose OpenAI-compatible APIs (/v1/chat/completions).
Workflow example
Inference API deployment: (1) define endpoint: POST /v1/chat/completions, (2) input format: {model, messages, temperature, max_tokens, ...}, (3) output format: {choices: [{message: {content}}], usage: {prompt_tokens, completion_tokens}}, (4) authentication: API keys or JWT tokens, (5) rate limiting: tokens per minute per user — enforced at the API gateway, (6) logging: log every request/response pair (minus PII) for evaluation and debugging.