How to build a multi-tenant AI metrics dashboard in Grafana with row-level security

What this does

This guide creates a single Grafana dashboard that serves multiple tenants with data isolation. Each tenant sees only their own AI usage metrics — token consumption, latency, error rates, and cost — without access to other tenants' data. The isolation is enforced at the data source level through templated queries that filter on a tenant_id label, combined with Grafana's data source permissions and variable scoping.

Steps

1. Label all AI service metrics with a tenant dimension. In the agent's metric instrumentation:

   requests_total = Counter("ai_requests_total", "API requests", ["tenant", "model", "status"])
   requests_total.labels(tenant=tenant_id, model=model_name, status="success").inc()
   

2. Create a Grafana tenant variable. In the dashboard settings, add a variable of type "Custom":

   • Name: tenant
   • Custom values: tenant_a,tenant_b,tenant_c
   • Enable "Multi-value" and "Include All option" Set this variable as the dashboard filter.

3. For strict data isolation, replace the custom variable with a query variable that reads tenants from an external source. Use a restricted data source that only returns the current tenant based on the authenticated user.

4. Apply the tenant filter to every panel query. For a token usage panel:

   sum by (model) (rate(ai_token_input_total{tenant=~"$tenant"}[5m]))
   

   The $tenant variable interpolates into the query, filtering data to the selected tenant(s).

5. Configure dashboard row permissions. In Grafana Enterprise, use dashboard permissions to restrict which rows a team can view. Create a separate row for admin-level panels (global aggregate metrics) and a user-scoped row for per-tenant panels.

6. Set up folder-level access control. Place the multi-tenant dashboard in a folder with restricted viewer access for each tenant team. Navigate to the folder settings > Permissions and add each team with "View" access.

7. Create a separate "admin overview" dashboard in an admin-only folder that queries all tenants without a tenant filter:

   sum by (tenant, model) (rate(ai_requests_total[5m]))
   

8. Test isolation. Log in as a tenant user, navigate to the dashboard, and confirm the tenant filter only shows data for that tenant's label value. Verify that manually removing the filter from the URL does not reveal other tenants' data (enforced at data source level via proxy).

Verification

curl -s -H "X-Grafana-Org-Id: 2" "http://localhost:3000/api/ds/query?expr=ai_requests_total" | jq '.data.result[] | .metric.tenant' | sort -u

Expected output: only the current tenant's identifier (confirmed isolation).

Common failures

• Tenant labels not present on all metrics — verify every Counter, Histogram, and Gauge in the agent code includes the tenant label. Metrics missing this label will aggregate across tenants, leaking data. Use promtool check metrics or manually inspect /metrics output.
• Grafana variable shows all tenants regardless of user — custom variable values are static and visible to all. Switch to a query-based variable that returns only the current user's tenant via a restricted data source.
• High query latency on admin panels — filtering with =~"$tenant" across hundreds of tenants produces expensive regex matching. Use exact match ="$tenant" when only one tenant is selected, and limit admin aggregate queries to a lower resolution (e.g., 5m instead of 30s).

Related guides

• Grafana dashboards for AI agent queue depth and wait times
• Grafana panels for vLLM throughput monitoring
• Resource quotas and limit ranges for AI namespaces