How to set up centralized logging for distributed AI agents using the ELK stack

What this does

This guide deploys an Elasticsearch, Logstash, Kibana (ELK) stack that aggregates structured logs from multiple AI agent instances into a single searchable index. Each agent writes JSON log lines to local files, Filebeat ships them to Logstash for enrichment and filtering, and Logstash indexes them into Elasticsearch. Operators can then use Kibana to search across all agents by correlation ID, error class, or model name.

Steps

1. Increase the virtual memory limit and persist the setting:

   sudo sysctl -w vm.max_map_count=262144
   echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
   

2. Create a docker-compose.yml for the ELK stack:

   version: "3.8"
   services:
     elasticsearch:
       image: docker.elastic.co/elasticsearch/elasticsearch:8.14.0
       environment:
         - discovery.type=single-node
         - xpack.security.enabled=false
       ports: ["9200:9200"]
       volumes: ["esdata:/usr/share/elasticsearch/data"]
     logstash:
       image: docker.elastic.co/logstash/logstash:8.14.0
       ports: ["5044:5044"]
       volumes: ["./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"]
     kibana:
       image: docker.elastic.co/kibana/kibana:8.14.0
       ports: ["5601:5601"]
       environment:
         - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
   volumes:
     esdata:
   

3. Create logstash.conf with a pipeline that parses agent JSON logs and enriches them:

   input { beats { port => 5044 } }
   filter {
     json { source => "message" }
     mutate { add_field => { "environment" => "production" } }
   }
   output { elasticsearch { hosts => ["elasticsearch:9200"] index => "ai-agents-%{+YYYY.MM.dd}" } }
   

4. Start the stack:

   docker compose up -d
   

   Expected output: three containers starting, confirmed with docker compose ps showing all services as healthy.

5. Install Filebeat on each AI agent host or add as a sidecar container:

   filebeat:
     image: docker.elastic.co/beats/filebeat:8.14.0
     volumes:
       - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
       - /var/log/ai-agent:/var/log/ai-agent:ro
   

6. Configure filebeat.yml to ship agent logs:

   filebeat.inputs:
     - type: log
       paths: /var/log/ai-agent/*.log
       json.keys_under_root: true
   output.logstash:
     hosts: ["logstash:5044"]
   

7. Restart services and verify logs flow. In Kibana at http://localhost:5601, navigate to Discover and create an index pattern for ai-agents-*. Set @timestamp as the time field.

8. Generate a test log from an agent instance and confirm it appears:

   docker compose exec agent python -c "import logging; logging.getLogger('ai-agent').info('test_event', extra={'type': 'manual_test'})"
   

Verification

curl -s http://localhost:9200/ai-agents-*/_count | jq '.count'

Expected output: an integer >= 0 (once logs are flowing).

Common failures

• Elasticsearch exits with code 78 — vm.max_map_count is too low. Set to 262144 and restart the container.
• Logstash cannot connect to Elasticsearch — if using a non-Docker Logstash, verify Elasticsearch is reachable at http://elasticsearch:9200. Inside Docker Compose, ensure both services share the same network.
• Filebeat does not ship logs — verify the agent log path is mounted read-only into the Filebeat container. Use docker compose exec filebeat filebeat test output to validate the connection.
• Kibana Discover shows no results — check the index pattern's time field. Agent logs must have a @timestamp field in ISO 8601 format.

Related guides

• Track AI agent conversation state with structured logging and correlation IDs
• Build a structured prompt/response logging pipeline with Fluentd
• Set up a multi-service AI logging stack with Docker Compose