What this does

This guide deploys a containerized logging stack comprising Grafana Loki (log aggregation), Promtail (log collection), and Grafana (visualization) alongside existing AI services in Docker Compose. Every AI container outputs structured JSON logs to stdout, Promtail tails the Docker log files and ships them to Loki, and Grafana provides a unified query interface across all services. This replaces scattered docker compose logs debugging with a centralized, searchable log platform.

Steps

Create the Loki configuration file loki-config.yml:

auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
  filesystem:
    directory: /loki/chunks

Create the Promtail configuration file promtail-config.yml:

server:
  http_listen_port: 9080
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_label_com_docker_compose_service]
        target_label: service

Add Loki, Promtail, and Grafana to the Docker Compose file:

services:
  loki:
    image: grafana/loki:2.9.0
    ports: ["3100:3100"]
    volumes: ["./loki-config.yml:/etc/loki/local-config.yaml", "loki_data:/loki"]
    command: -config.file=/etc/loki/local-config.yaml
  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - "./promtail-config.yml:/etc/promtail/config.yml"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
    command: -config.file=/etc/promtail/config.yml
  grafana:
    image: grafana/grafana:10.4.0
    ports: ["3000:3000"]
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
    volumes: ["grafana_data:/var/lib/grafana"]
volumes:
  loki_data:
  grafana_data:

Reconfigure the AI services to emit JSON logs. For Python services, update the logging configuration:

import logging, json, sys
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter('{"level":"%(levelname)s","service":"ai-agent","message":"%(message)s","timestamp":"%(asctime)s"}'))
logging.getLogger().addHandler(handler)

Tag Docker Compose service logs so Promtail can identify them. Add labels to each AI service:
```
labels:
  logging: "promtail"
  com.docker.compose.service: "ai-agent"
```
Start the entire stack and verify all services:
```
docker compose up -d && docker compose ps
```
Add Loki as a data source in Grafana. In Grafana at http://localhost:3000, navigate to Connections > Data Sources > Add data source > Loki. Set URL to http://loki:3100 and click "Save & test."
Query logs across services. In Grafana Explore, select the Loki data source and enter:
```
{service=~"ai-agent|inference"}
```
Expected: log lines from both services, color-coded by service.

Verification

curl -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="docker"}' | jq '.data.result | length'

Expected output: an integer >= 1 (stream count, confirming log ingestion).

Common failures

Promtail shows "permission denied" on Docker socket — add user: "0:0" to the Promtail service definition or add the Promtail container user to the docker group on the host.
Loki ingester returns 503 — the ingester ring needs a few seconds to initialize. Wait 10 seconds and retry.
No logs appear in Grafana — Promtail may be targeting the wrong directory for container logs. Verify the mount: /var/lib/docker/containers:/var/lib/docker/containers:ro exists and contains subdirectories.

How to set up a multi-service AI logging stack with Docker Compose

What this does

Steps

Verification

Common failures

Related guides