13. Cross-Tier Monitoring

Cross-tier monitoring provides unified observability across local and cloud inference paths. Without thorough visibility, teams operate with blind spots—local performance issues go undetected until they impact users, cloud degradation goes unnoticed until costs spike, and routing decisions lack empirical grounding.

Effective monitoring aggregates metrics from heterogeneous sources. Local inference servers expose Prometheus-format metrics. Cloud providers offer API monitoring dashboards and log exports. The monitoring layer normalizes these disparate sources into consistent time-series data.

from prometheus_client import Counter, Histogram, Gauge

# Metrics definitions
request_total = Counter(
    'ai_gateway_requests_total',
    'Total requests processed',
    ['provider', 'model', 'status']
)

request_latency = Histogram(
    'ai_gateway_request_duration_seconds',
    'Request latency in seconds',
    ['provider', 'model']
)

token_usage = Counter(
    'ai_gateway_tokens_total',
    'Tokens consumed',
    ['provider', 'model', 'direction']
)

local_queue_depth = Gauge(
    'ai_gateway_local_queue_depth',
    'Pending local inference requests',
    ['model', 'instance']
)

def record_request(provider: str, model: str, 
                   latency: float, tokens: int, 
                   success: bool):
    status = 'success' if success else 'failure'
    request_total.labels(provider, model, status).inc()
    request_latency.labels(provider, model).observe(latency)
    token_usage.labels(provider, model, 'output').inc(tokens)

Alerting configuration requires balancing sensitivity and noise. Critical alerts for complete provider failures should page immediately. Warning alerts for degraded performance or elevated error rates should notify but not page. Trend alerts for capacity planning—approaching local GPU utilization thresholds—should feed into operational dashboards without triggering immediate response.

Dashboard design groups metrics by operational concern: latency percentiles for SLA tracking, error rates for reliability assessment, token costs for budget monitoring, and queue depths for capacity planning. Grafana dashboards provide flexible visualization across these dimensions.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.