18. Rollback Strategies

Chapter 18 of 24 · 20 min

KEY INSIGHT

Rollback automation must be tested as frequently as deployment automation; procedures that fail when invoked under pressure defeat their purpose entirely. ### Kubernetes Rollback ```bash # Immediate rollback to previous revision kubectl rollout undo deployment/inference-server -n production # Rollback to specific revision kubectl rollout history deployment/inference-server -n production kubectl rollout undo deployment/inference-server \ --to-revision=3 -n production # Watch rollback progress kubectl rollout status deployment/inference-server -n production --timeout=300s ``` ### Automated Rollback with Argo Rollouts ```yaml # analysis-template.yaml (abort criteria) spec: metrics: - name: error-rate-critical interval: 2m failureCondition: result[0] > 0.05 provider: prometheus: query: | sum(rate(inference_errors_total[2m])) / sum(rate(inference_requests_total[2m])) > 0.05 - name: latency-critical interval: 2m failureCondition: result[0] > 3000 provider: prometheus: query: | histogram_quantile(0.95, inference_latency_seconds) > 3000 - name: throughput-degraded interval: 5m failureCondition: result[0] < 0.5 * 100 provider: prometheus: query: | rate(inference_requests_total[5m]) < 50 ``` ### Grace Period Configuration Configure appropriate rollback windows that allow post-deployment observation: ```yaml spec: strategy: canary: analysis: templates: - templateName: inference-analysis startingStep: 2 limit: 2 args: - name: service-name value: inference-rollout ``` ### Rolling Update Rollback ```python # scripts/rollback_handler.py import kubernetes from kubernetes.client import AppsV1Api def check_rollback_criteria(deployment_name: str, namespace: str) -> bool: """Evaluate whether automatic rollback should trigger.""" api = AppsV1Api() rollout = api.read_namespaced_deployment( name=deployment_name, namespace=namespace ) # Check for excessive error rates error_rate = get_error_rate(deployment_name, namespace) if error_rate > 0.05: return True # Check for timeout conditions last_update = rollout.status.conditions[-1].last_transition_time elapsed = datetime.now(timezone.utc) - last_update if elapsed > timedelta(minutes=30): if rollout.status.available_replicas < rollout.spec.replicas: return True return False def execute_rollback(deployment_name: str, namespace: str): """Execute rollback procedure.""" api = AppsV1Api() # Rollback to previous version api.patch_namespaced_deployment( name=deployment_name, namespace=namespace, body={"spec": {"template": {"metadata": {"annotations": { "kubectl.kubernetes.io/restartedAt": datetime.now().isoformat() }}}}} ) # Log rollback event log_event( f"Automatic rollback executed for {deployment_name}", severity="critical" ) ```

Despite thorough testing, production incidents occasionally require reverting to previous model versions or configurations. Automated rollback mechanisms minimize mean time to recovery by executing pre-defined reversal procedures without human intervention during high-stress incidents.

EXERCISE

Deploy a model to a Kubernetes cluster, then simulate a failure by introducing faulty behavior. Configure automated rollback triggers based on error rate thresholds. Verify that the cluster automatically reverts to the previous revision within the configured timeout window.