What this does

This guide deploys an Elasticsearch, Logstash, Kibana (ELK) stack that aggregates structured logs from multiple AI agent instances into a single searchable index. Each agent writes JSON log lines to local files, Filebeat ships them to Logstash for enrichment and filtering, and Logstash indexes them into Elasticsearch. Operators can then use Kibana to search across all agents by correlation ID, error class, or model name.

Steps

Increase the virtual memory limit and persist the setting:

sudo sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf

Create a docker-compose.yml for the ELK stack:

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.14.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports: ["9200:9200"]
    volumes: ["esdata:/usr/share/elasticsearch/data"]
  logstash:
    image: docker.elastic.co/logstash/logstash:8.14.0
    ports: ["5044:5044"]
    volumes: ["./logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"]
  kibana:
    image: docker.elastic.co/kibana/kibana:8.14.0
    ports: ["5601:5601"]
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
volumes:
  esdata:

Create logstash.conf with a pipeline that parses agent JSON logs and enriches them:

input { beats { port => 5044 } }
filter {
  json { source => "message" }
  mutate { add_field => { "environment" => "production" } }
}
output { elasticsearch { hosts => ["elasticsearch:9200"] index => "ai-agents-%{+YYYY.MM.dd}" } }

Start the stack:
```
docker compose up -d
```
Expected output: three containers starting, confirmed with docker compose ps showing all services as healthy.

Install Filebeat on each AI agent host or add as a sidecar container:

filebeat:
  image: docker.elastic.co/beats/filebeat:8.14.0
  volumes:
    - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
    - /var/log/ai-agent:/var/log/ai-agent:ro

Configure filebeat.yml to ship agent logs:

filebeat.inputs:
  - type: log
    paths: /var/log/ai-agent/*.log
    json.keys_under_root: true
output.logstash:
  hosts: ["logstash:5044"]

Restart services and verify logs flow. In Kibana at http://localhost:5601, navigate to Discover and create an index pattern for ai-agents-*. Set @timestamp as the time field.

Generate a test log from an agent instance and confirm it appears:

docker compose exec agent python -c "import logging; logging.getLogger('ai-agent').info('test_event', extra={'type': 'manual_test'})"

Verification

curl -s http://localhost:9200/ai-agents-*/_count | jq '.count'

Expected output: an integer >= 0 (once logs are flowing).

Common failures

Elasticsearch exits with code 78 — vm.max_map_count is too low. Set to 262144 and restart the container.
Logstash cannot connect to Elasticsearch — if using a non-Docker Logstash, verify Elasticsearch is reachable at http://elasticsearch:9200. Inside Docker Compose, ensure both services share the same network.
Filebeat does not ship logs — verify the agent log path is mounted read-only into the Filebeat container. Use docker compose exec filebeat filebeat test output to validate the connection.
Kibana Discover shows no results — check the index pattern's time field. Agent logs must have a @timestamp field in ISO 8601 format.

How to set up centralized logging for distributed AI agents using the ELK stack

What this does

Steps

Verification

Common failures

Related guides