HOW-TO · OPS

How to create Grafana dashboards for AI agent queue depth and wait times

intermediate30 minBy Eruo Fredoline
Target environment
Ubuntu 24.04 · Ollama 0.4.x
PREREQUISITES

Grafana running, Prometheus as data source, queue metrics

What this does

This guide builds a Grafana dashboard that displays real-time queue metrics for AI agent workloads: current queue depth, average wait time, processing rate, and oldest message age. The dashboard helps operators identify backpressure before it causes request timeouts. All panels query Prometheus metrics exposed by Redis or RabbitMQ exporters.

Steps

  1. Verify queue metrics are available in Prometheus. For Redis-backed queues:

    curl http://localhost:9090/api/v1/query?query=redis_list_length

    Expected output: JSON with status: "success" and a non-null value.

  2. Log into Grafana and create a new dashboard. Click the + icon in the sidebar and select "New Dashboard", then "Add visualization".

  3. Add a time-series panel for queue depth. Set the query to:

    ai_request_queue_depth

    Configure the panel title as "Request Queue Depth", unit as "short", and set Y-axis minimum to 0.

  4. Add a stat panel for current queue depth. Use the same metric with an instant query:

    ai_request_queue_depth

    Set the stat style to display the current (last) value with a threshold of green < 10, yellow 10-50, red > 50.

  5. Add a time-series panel for average wait time. Use the Prometheus rate function:

    rate(ai_request_queue_wait_seconds_sum[5m]) / rate(ai_request_queue_wait_seconds_count[5m])

    Label the panel "Average Queue Wait Time" with unit "seconds".

  6. Add a gauge panel for the queue processing rate:

    rate(ai_request_processed_total[5m])

    Title it "Processing Rate (requests/sec)" and set unit to "requests/sec".

  7. Add an alert: create a dashboard alert rule on the queue depth panel with condition WHEN avg() OF query(A, 5m, now) IS ABOVE 100. Set the evaluation interval to 1 minute and configure a notification channel.

  8. Save the dashboard with a descriptive name, e.g., "AI Agent Queue Operations". Export the dashboard JSON for version control:

    curl -H "Authorization: Bearer <api-key>" http://localhost:3000/api/dashboards/uid/<uid> | jq '.dashboard' > queue-dashboard.json

Verification

curl -s "http://localhost:3000/api/dashboards/uid/ai-agent-queue" -H "Authorization: Bearer <token>" | jq '.dashboard.title'

Expected output: "AI Agent Queue Operations".

Common failures

  • Panels show "No data" — check that the metric name in the panel query exactly matches the metric exposed by the queue exporter. Use Prometheus's /api/v1/label/__name__/values to list available metrics.
  • Wait time panel returns NaN — the _sum and _count time series need at least one observation each. Trigger a few requests through the queue and re-check.
  • Dashboard alert does not fire — confirm the alert rule evaluation interval and ensure the notification channel (email, Slack, PagerDuty) is correctly configured under Alerting > Contact points.
  • Stat panel shows stale value — the instant query may be cached. Set the panel's "Max data points" to 1 and reduce the dashboard refresh interval to 10 seconds.

Related guides

← All how-to guidesCourses →