How to set up a multi-service AI logging stack with Docker Compose

What this does

This guide deploys a containerized logging stack comprising Grafana Loki (log aggregation), Promtail (log collection), and Grafana (visualization) alongside existing AI services in Docker Compose. Every AI container outputs structured JSON logs to stdout, Promtail tails the Docker log files and ships them to Loki, and Grafana provides a unified query interface across all services. This replaces scattered docker compose logs debugging with a centralized, searchable log platform.

Steps

1. Create the Loki configuration file loki-config.yml:

   auth_enabled: false
   server:
     http_listen_port: 3100
   ingester:
     lifecycler:
       ring:
         kvstore:
           store: inmemory
   schema_config:
     configs:
       - from: 2024-01-01
         store: tsdb
         object_store: filesystem
         schema: v13
         index:
           prefix: index_
           period: 24h
   storage_config:
     tsdb_shipper:
       active_index_directory: /loki/index
     filesystem:
       directory: /loki/chunks
   

2. Create the Promtail configuration file promtail-config.yml:

   server:
     http_listen_port: 9080
   clients:
     - url: http://loki:3100/loki/api/v1/push
   scrape_configs:
     - job_name: docker
       docker_sd_configs:
         - host: unix:///var/run/docker.sock
       relabel_configs:
         - source_labels: [__meta_docker_container_name]
           target_label: container
         - source_labels: [__meta_docker_container_label_com_docker_compose_service]
           target_label: service
   

3. Add Loki, Promtail, and Grafana to the Docker Compose file:

   services:
     loki:
       image: grafana/loki:2.9.0
       ports: ["3100:3100"]
       volumes: ["./loki-config.yml:/etc/loki/local-config.yaml", "loki_data:/loki"]
       command: -config.file=/etc/loki/local-config.yaml
     promtail:
       image: grafana/promtail:2.9.0
       volumes:
         - "./promtail-config.yml:/etc/promtail/config.yml"
         - "/var/run/docker.sock:/var/run/docker.sock"
         - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
       command: -config.file=/etc/promtail/config.yml
     grafana:
       image: grafana/grafana:10.4.0
       ports: ["3000:3000"]
       environment:
         - GF_AUTH_ANONYMOUS_ENABLED=true
       volumes: ["grafana_data:/var/lib/grafana"]
   volumes:
     loki_data:
     grafana_data:
   

4. Reconfigure the AI services to emit JSON logs. For Python services, update the logging configuration:

   import logging, json, sys
   handler = logging.StreamHandler(sys.stdout)
   handler.setFormatter(logging.Formatter('{"level":"%(levelname)s","service":"ai-agent","message":"%(message)s","timestamp":"%(asctime)s"}'))
   logging.getLogger().addHandler(handler)
   

5. Tag Docker Compose service logs so Promtail can identify them. Add labels to each AI service:

   labels:
     logging: "promtail"
     com.docker.compose.service: "ai-agent"
   

6. Start the entire stack and verify all services:

   docker compose up -d && docker compose ps
   

7. Add Loki as a data source in Grafana. In Grafana at http://localhost:3000, navigate to Connections > Data Sources > Add data source > Loki. Set URL to http://loki:3100 and click "Save & test."

8. Query logs across services. In Grafana Explore, select the Loki data source and enter:

   {service=~"ai-agent|inference"}
   

   Expected: log lines from both services, color-coded by service.

Verification

curl -s "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="docker"}' | jq '.data.result | length'

Expected output: an integer >= 1 (stream count, confirming log ingestion).

Common failures

• Promtail shows "permission denied" on Docker socket — add user: "0:0" to the Promtail service definition or add the Promtail container user to the docker group on the host.
• Loki ingester returns 503 — the ingester ring needs a few seconds to initialize. Wait 10 seconds and retry.
• No logs appear in Grafana — Promtail may be targeting the wrong directory for container logs. Verify the mount: /var/lib/docker/containers:/var/lib/docker/containers:ro exists and contains subdirectories.

Related guides

• Centralized logging for distributed AI agents using the ELK stack
• Build a structured prompt/response logging pipeline with Fluentd
• Track AI agent conversation state with structured logging and correlation IDs