06. Latency-Aware Routing

Latency represents a critical quality dimension for interactive AI applications. Users abandon slow responses, and downstream systems impose timeout constraints. Latency-aware routing evaluates backend response time predictions and routes requests to backends likely to meet performance requirements.

Backend characterization establishes latency expectations for each model and hardware combination. Historical measurements reveal typical response distributions. Model size correlates strongly with inference duration. Hardware specifications (GPU memory, vRAM bandwidth) predict capability ceilings. Operators build reference tables mapping backend configurations to expected latency ranges.

Predictive routing incorporates real-time queue status into delay forecasts. Traffic spikes temporarily inflate queue wait times. Concurrency limits constrain parallel request processing. By combining historical expectations with current load measurements, the router estimates total request duration before committing to a backend.

json
{
  "backend_profiles": {
    "local-llama-70b": {
      "model": "llama-70b-instruct",
      "hardware": "H100-80GB",
      "predictors": {
        "base_latency_p50_ms": 800,
        "base_latency_p95_ms": 1500,
        "tokens_per_second": 45,
        "max_concurrent": 4
      },
      "current_load": {
        "queue_depth": 2,
        "active_requests": 1
      }
    },
    "cloud-gpt4": {
      "model": "gpt-4-turbo",
      "provider": "openai",
      "predictors": {
        "base_latency_p50_ms": 2000,
        "base_latency_p95_ms": 8000,
        "tokens_per_second": 150,
        "max_concurrent": 100
      },
      "current_load": {
        "queue_depth": 0,
        "active_requests": 15
      }
    }
  },
  "routing_thresholds": {
    "max_acceptable_latency_ms": 5000,
    "p95_target_multiplier": 1.5,
    "minimum_backend_confidence": 0.8
  }
}

Service level objectives codify latency targets into enforceable constraints. Interactive applications may target sub-second responses. Batch processing tolerates multi-minute delays. Document analysis workflows sit between these extremes. The router enforces these objectives by excluding backends predicted to miss targets.

Adaptive thresholds adjust based on request characteristics. Simple queries face tighter constraints than complex analysis. Time-sensitive applications receive priority acceleration. Geographic distribution introduces latency variance that regional routing mitigates.

Timeout orchestration coordinates deadlines across system boundaries. Client timeouts signal acceptable duration limits. Backend timeouts prevent resource exhaustion on hung requests. Retry policies account for single-attempt latency budgets. Cascading timeout propagation prevents optimistic time budgeting that results in cascading failures.