Inference API
An inference API is a programmatic interface that accepts input data (like a prompt) and returns a model's output (like generated text) over HTTP. Operators encounter it when they expose a local model to other applications—for example, using llama.cpp's built-in HTTP server or vLLM's OpenAI-compatible endpoint. The API handles request queuing, tokenization, and generation, returning results as JSON. It matters because running a local inference API avoids sending data to third-party services and allows integration with tools like SillyTavern, Open WebUI, or custom scripts.
Deeper dive
Inference APIs standardize how clients interact with LLMs. The most common protocol is the OpenAI Chat Completions API format, which many local runtimes (vLLM, llama.cpp, Ollama, LM Studio) implement. This means tools built for OpenAI's API can often be pointed at a local endpoint by changing the base URL. The API typically handles batching, streaming (server-sent events), and parameter passing (temperature, top_p, max_tokens). Operators may also encounter REST endpoints for embeddings, tokenization, or model listing. Performance considerations include request throughput (requests per second), latency per request, and concurrent user handling—vLLM uses continuous batching to maximize GPU utilization, while llama.cpp's server is simpler but may queue requests.
Practical example
An operator running llama.cpp can start an inference API with ./server -m model.gguf --host 0.0.0.0 --port 8080. Then a Python script can send a request using requests.post('http://localhost:8080/v1/chat/completions', json={'model': 'model', 'messages': [{'role': 'user', 'content': 'Hello'}]}). The API returns JSON with the assistant's reply. On an RTX 3090, this setup can handle ~10-20 concurrent requests with ~50 tok/s each, depending on model size and quantization.
Workflow example
In LM Studio, enabling the built-in inference API is a checkbox in the settings—it starts a local server on port 1234. Then in Open WebUI, you configure the connection to http://localhost:1234/v1 and select the model. All chat requests go through the API, and you can monitor token generation speed in the LM Studio logs. Similarly, vLLM's --api-key flag secures the endpoint, and operators can use curl to test: curl http://localhost:8000/v1/completions -H "Authorization: Bearer mykey" -d '{"model": "model", "prompt": "Hello"}'.
Reviewed by Fredoline Eruo. See our editorial policy.