How to create SLO burn rate alerts for AI agent availability

What this does

This guide implements multiwindow, multi-burn-rate alerts for AI agent service level objectives (SLOs). Using the approach described in the Site Reliability Engineering workbook, it defines alerts that fire when the error budget is being consumed faster than the SLO allows — with separate short-window (page) and long-window (ticket) alert severities. This catches both sudden outages and slow degradations.

Steps

1. Define the SLI as a Prometheus recording rule. Create slo-rules.yml:

   groups:
     - name: ai-agent-slo
       rules:
         - record: sli:ai_agent_availability_ratio
           expr: |
             sum(rate(ai_requests_total{status="success"}[5m]))
             /
             sum(rate(ai_requests_total[5m]))
   

2. Define the burn rate alerting thresholds. For a 99.9% SLO with a 30-day window, the error budget is 0.1%. A burn rate of 14.4x consumes the full error budget in 1 hour (critical page); 1x consumes it over 30 days (warning ticket).

3. Add the multiwindow burn rate alert rules:

   - alert: AIAgentHighErrorBurnRate
     expr: |
       (1 - sli:ai_agent_availability_ratio)
       > (14.4 * 0.001)
       and
       (1 - sli:ai_agent_availability_ratio offset 5m)
       > (14.4 * 0.001)
       and
       (1 - sli:ai_agent_availability_ratio offset 1h)
       > (14.4 * 0.001)
     for: 2m
     labels:
       severity: page
     annotations:
       summary: "AI agent error budget burning at >14x rate"
   

4. Add a low-severity burn rate alert for slow degradation:

   - alert: AIAgentLowErrorBurnRate
     expr: |
       (1 - sli:ai_agent_availability_ratio)
       > (1.0 * 0.001)
       and
       (1 - sli:ai_agent_availability_ratio offset 1h)
       > (1.0 * 0.001)
     for: 10m
     labels:
       severity: ticket
     annotations:
       summary: "AI agent error budget burning at >1x rate over 6 hours"
   

5. Validate the rules files:

   promtool check rules slo-rules.yml
   

   Expected output: SUCCESS: 1 rules found.

6. Load rules into Prometheus by adding to prometheus.yml:

   rule_files:
     - "slo-rules.yml"
   

7. Reload Prometheus and verify in the Rules page at http://localhost:9090/rules. Confirm both alerts appear as "inactive".

8. Simulate an error burst to test the alert fires:

   for _ in range(100):
       inc_ai_request(status="error")
   

   Wait 5 minutes, then check Alertmanager at http://localhost:9093/#/alerts.

Verification

curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="ai-agent-slo") | .rules[] | .name'

Expected output: "AIAgentHighErrorBurnRate" and "AIAgentLowErrorBurnRate".

Common failures

• Alerts never fire — the recording rule requires at least 5 minutes of data. Confirm the metric ai_requests_total exists with curl http://localhost:9090/api/v1/label/__name__/values | jq '.data[]'.
• Alert fires immediately after deployment — the short window (5m) combined with low traffic produces high volatility. Increase the "for" duration or use min_over_time to smooth the signal.
• False positives during maintenance windows — add a silencing rule in Alertmanager or exclude the maintenance window in the recording rule using unless on() (maintenance_window_active == 1).

Related guides

• Monitor AI agent error rates by error classification
• Set up Prometheus alerting rules for AI service degradation
• Instrument a Python FastAPI AI service with Prometheus metrics