01. Production Mindset
The transition from development experimentation to production deployment requires a fundamental shift in how systems get designed, monitored, and maintained. Development environments tolerate restarts, manual intervention, and incomplete error handling. Production environments demand reliability, observability, and automation for common failure scenarios.
A production mindset treats infrastructure as code with version control, peer review, and automated testing. Configuration drifts represent bugs. Manual deployments on production systems constitute incidents. Every running service requires a clear owner, a monitoring dashboard, and documented rollback procedures.
Key differences between development and production thinking include startup time expectations, graceful degradation requirements, and the distinction between errors that block operations versus errors that generate informative logs. Development might tolerate a five-minute startup. Production often requires sub-minute initialization with readiness probes confirming service availability before traffic routing.
Incident response procedures must exist before incidents occur. Runbooks document the steps for common failure modes: GPU memory exhaustion, OOM kills, network partition between services, and model loading failures. The on-call engineer should not discover troubleshooting steps during an outage.
Health checks serve two purposes in production systems. Liveness probes indicate when a process should restart. Readiness probes indicate when a service can receive traffic. Misconfiguring these probes causes cascading failures where restarted pods immediately receive requests before finishing initialization.
Configuration management requires separation between environment parameters and application code. Environment variables drive behavior without image rebuilds. Secrets require special handling with encryption at rest, encryption in transit, and access control limiting which pods can read which secrets.
Define monitoring requirements for an AI inference endpoint: latency percentiles, error rates, GPU utilization, and memory pressure. Write the alerting rules in Prometheus format for each metric with appropriate severity levels and notification channels.
# Latency alert: 95th percentile exceeds 2 seconds
- alert: InferenceLatencyHigh
expr: histogram_quantile(0.95, rate(inference_request_duration_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "Inference latency above 2s at 95th percentile"