04. Chat Completions Endpoint

Chapter 4 of 18 · 20 min

KEY INSIGHT

The chat completions endpoint is the workhorse of AI APIs. Implementing it correctly means handling message formatting, inference calls, token counting, and error propagation in a single request-response cycle that must complete within tight time constraints. ### Implementation Walkthrough Start with the Pydantic model that mirrors the OpenAI request format. ```python from pydantic import BaseModel, Field from typing import Literal class Message(BaseModel): role: Literal["system", "user", "assistant"] content: str class CompletionRequest(BaseModel): model: str messages: list[Message] temperature: float = Field(default=0.7, ge=0, le=2) max_tokens: int = Field(default=256, gt=0) stream: bool = False top_p: float | None = None frequency_penalty: float | None = None presence_penalty: float | None = None ``` Field validators enforce constraints like temperature range. The request model handles validation before any inference code runs. ### Building the Response ```python import time class CompletionResponse(BaseModel): id: str object: str = "chat.completion" created: int model: str choices: list usage: dict def create_response(model: str, message: str, tokens: int) -> dict: return { "id": f"chatcmpl-{random_id()}", "object": "chat.completion", "created": int(time.time()), "model": model, "choices": [{ "index": 0, "message": {"role": "assistant", "content": message}, "finish_reason": "stop" }], "usage": { "prompt_tokens": count_tokens(message), "completion_tokens": tokens, "total_tokens": count_tokens(message) + tokens } } ``` The response follows the exact OpenAI schema. Token counting requires a tokenizer that matches the model being served. Using the wrong tokenizer produces incorrect usage statistics. ### Handling Inference ```python async def generate(request: CompletionRequest): try: # Convert messages to prompt format expected by model prompt = format_messages(request.messages) # Call inference service result = await inference_client.generate( prompt=prompt, max_tokens=request.max_tokens, temperature=request.temperature ) return create_response(request.model, result.text, result.tokens) except ModelNotFoundError: raise HTTPException(status_code=404, detail="Model not found") except InferenceTimeoutError: raise HTTPException(status_code=504, detail="Inference timeout") ``` Error handling converts internal exceptions into appropriate HTTP status codes. Clients should never receive a 500 error with a raw traceback.

EXERCISE

Implement the chat completions endpoint using a mock inference function that returns static text after a 500ms delay. Measure the total response time and verify the usage statistics are included in the response.