Skip to main content

Logging with ELK and fluentd

Install the ELK stack:​

helm upgrade --install elk-elasticsearch elastic/elasticsearch -f elastic_values.yaml -n logging --create-namespace
helm upgrade --install elk-logstash elastic/logstash -f logstash_values.yaml -n logging
helm upgrade --install elk-kibana elastic/kibana -f kibana_values.yaml -n logging

# optionally install filebeat if you plan to use filebeat instead of fluentd
helm upgrade --install elk-filebeat elastic/filebeat -f filebeat_values.yaml -n logging

Install fluentd daemonset, this can be found on github:​

kubectl apply -f fluentd-daemonset-elasticsearch.yaml

Alternatively, you can also deploy via the helm charts provided by fluentbit.

Setup ELK indexes​

  1. Check for available indices - port-forward the elasticsearch-master on port 9100 and visit the http://localhost:9200/_cat/indices, you should see the following logstash-* indices available:
      green  open .geoip_databases                rP6BifVQSuCv1XmctC0M_Q 1 0  40   0  38.4mb  38.4mb
      green  open .kibana_task_manager_7.17.3_001 p5Idg-xWTpCj4TWh6YpNrQ 1 0  17 543 123.6kb 123.6kb
      yellow open logstash-2022.10.10             nyG-OW_qRKCBertmmOwwyw 1 1 895   0 416.6kb 416.6kb  ◀─────┐
      green  open .apm-custom-link                jv3jzCztQUujEYwYv1iTIw 1 0   0   0    226b    226b        │      ┌───────────────┐
      green  open .apm-agent-configuration        NsZHlaeGSmSc7xSa8CGcOA 1 0   0   0    226b    226b        │      │   Logstash    │
      yellow open logstash-2022.10.07             cW3b1TJlROCwV2BKkzpt2Q 1 1 212   0  52.1kb  52.1kb  ◀─────┼──────│    Entries    │
      yellow open logstash-2022.10.08             yzU4pqq3QOyZkukcmGKpaw 1 1 172   0  43.6kb  43.6kb  ◀─────┤      └───────────────┘
      yellow open logstash-2022.10.09             n9GQnFB4RSWlWwkFG1848g 1 1 866   0 100.4kb 100.4kb  ◀─────┘
      green  open .kibana_7.17.3_001              BjXjQqXcRoiiGQg_zsrSrg 1 0  21   8   2.3mb   2.3mb
  2. Next port-forward the kibana dashboard on its default port 5601 and navigate to http://localhost:5601/app/management or choose "Stack Management" from left menu side bar. Screenshot 2022-10-10 at 3 46 50 PM
  3. Choose "Index Patterns" and click on "Create index Pattern"Screenshot 2022-10-10 at 3 49 07 PM
  4. Type the Name as logstash* and @timestamp for the Timestamp field and click on "Create index pattern"Screenshot 2022-10-10 at 3 50 13 PM
  5. Now you can navigate to http://localhost:5601/app/discover or click on "Discover" from the left sidebar menu and should start seeing your logs.image