What this does

This guide implements multiwindow, multi-burn-rate alerts for AI agent service level objectives (SLOs). Using the approach described in the Site Reliability Engineering workbook, it defines alerts that fire when the error budget is being consumed faster than the SLO allows — with separate short-window (page) and long-window (ticket) alert severities. This catches both sudden outages and slow degradations.

Steps

Define the SLI as a Prometheus recording rule. Create slo-rules.yml:

groups:
  - name: ai-agent-slo
    rules:
      - record: sli:ai_agent_availability_ratio
        expr: |
          sum(rate(ai_requests_total{status="success"}[5m]))
          /
          sum(rate(ai_requests_total[5m]))

Define the burn rate alerting thresholds. For a 99.9% SLO with a 30-day window, the error budget is 0.1%. A burn rate of 14.4x consumes the full error budget in 1 hour (critical page); 1x consumes it over 30 days (warning ticket).

Add the multiwindow burn rate alert rules:

- alert: AIAgentHighErrorBurnRate
  expr: |
    (1 - sli:ai_agent_availability_ratio)
    > (14.4 * 0.001)
    and
    (1 - sli:ai_agent_availability_ratio offset 5m)
    > (14.4 * 0.001)
    and
    (1 - sli:ai_agent_availability_ratio offset 1h)
    > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "AI agent error budget burning at >14x rate"

Add a low-severity burn rate alert for slow degradation:

- alert: AIAgentLowErrorBurnRate
  expr: |
    (1 - sli:ai_agent_availability_ratio)
    > (1.0 * 0.001)
    and
    (1 - sli:ai_agent_availability_ratio offset 1h)
    > (1.0 * 0.001)
  for: 10m
  labels:
    severity: ticket
  annotations:
    summary: "AI agent error budget burning at >1x rate over 6 hours"

Validate the rules files:
```
promtool check rules slo-rules.yml
```
Expected output: SUCCESS: 1 rules found.
Load rules into Prometheus by adding to prometheus.yml:
```
rule_files:
  - "slo-rules.yml"
```
Reload Prometheus and verify in the Rules page at http://localhost:9090/rules. Confirm both alerts appear as "inactive".
Simulate an error burst to test the alert fires:
```
for _ in range(100):
    inc_ai_request(status="error")
```
Wait 5 minutes, then check Alertmanager at http://localhost:9093/#/alerts.

Verification

curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="ai-agent-slo") | .rules[] | .name'

Expected output: "AIAgentHighErrorBurnRate" and "AIAgentLowErrorBurnRate".

Common failures

Alerts never fire — the recording rule requires at least 5 minutes of data. Confirm the metric ai_requests_total exists with curl http://localhost:9090/api/v1/label/__name__/values | jq '.data[]'.
Alert fires immediately after deployment — the short window (5m) combined with low traffic produces high volatility. Increase the "for" duration or use min_over_time to smooth the signal.
False positives during maintenance windows — add a silencing rule in Alertmanager or exclude the maintenance window in the recording rule using unless on() (maintenance_window_active == 1).

How to create SLO burn rate alerts for AI agent availability

What this does

Steps

Verification

Common failures

Related guides