Kubernetes is currently the industry’s de-facto standard for container orchestration. It provides the essential abstraction for efficiently managing large-scale containerized applications with declarative configurations, a deployment procedure, scaling, and self-healing features.
The Kubernetes ecosystem has grown thanks to additional integrations created by the community, such as logging and monitoring. Kubernetes' logging mechanism is an essential tool for managing and monitoring infrastructure and services. The ability to monitor faults and even fine-tune the performance of the containers that host the apps makes logs useful in Kubernetes. Fluentd, a logging agent, handles log collecting, parsing, and distribution in the background.
Fluentd Kubernetes - Table of Content
The open-source Kubernetes container orchestration engine automates the setup, expansion, and management of containerized applications. The organisation behind the open-source project is the Cloud Native Computing Foundation.
Before getting started, it’s better to have a basic understanding of the following Kubernetes concepts:
|Looking forward to becoming a master in Containerization Courses? Check out the "Kubernetes Training" and get certified today.|
Tracking cluster activity and fixing problems are made much easier by the logs. In Kubernetes, there are three main logging levels: basic, node-level, and cluster-level.
To implement cluster-level logging, a node-level logging agent can be installed on each node. To expose logs or push logs to a backend, a specialized tool called a logging agent is employed. The logging agent is often a container that has access to a directory holding log files from all of the application containers on that node.
Because the logging agent must be running on every node, it is recommended that you run the agent as a DaemonSet.
Node-level logging only creates one agent, and the programs already executing on the node do not need to be changed.
Although there is no set format, containers can write to stderr and stdout. A gathers and sends these logs for aggregate.
You can use a sidecar container in any of the following ways:
The cluster-logging feature of Kubernetes does not expose or push logs directly from every application.
Now that we covered the fundamentals of logging, let’s explore the crucial component that can make logging tasks easier: Fluentd
|Would you like to ace Kubernetes job interviews? Top Kubernetes Interview Questions from MindMajix are exclusively for you!|
Fluentd is a free and open-source data collector, which lets you unify the data. For better data utilization and interpretation, you can integrate data collection and consumption with Fluentd.
The Kubernetes container log files will be put up on Fluentd nodes, which will then filter and change the log data before sending it to the Elasticsearch cluster for indexing and storage.
Fluentd is a useful log aggregator. It was made in Ruby and is extremely scalable. For the vast majority of small- to medium-sized installations, Fluentd is speedy and only needs a little number of resources.
Fluentd gathers logs from a predetermined set of sources, processes them (converting them into a structured data format), and then passes them on to other services like Elasticsearch, object storage, etc. in order to send logs to these other services. Fluentd supports more than 300 log storage and analysis services, which makes it incredibly flexible when it comes to integrations.
Fluentd receives data from a variety of sources.
For Kubernetes, a DaemonSet ensures that all (or a portion of) nodes are running a copy of a pod. To address the issue of log collection, we'll employ a Fluentd DaemonSet.
Step 1: Grant Permissions to Fluentd
Since Fluentd will be collecting logs from user apps as well as cluster components like
kube-scheduler, we must grant it some access.
Giving the future Fluentd DaemonSet a name is the first thing we must do. Let's build a new ServiceAccount in the Kube-system namespace to deploy Fluentd
apiVersion: vl kind: ServiceAccount metadata: name: fluentd namespace: kube-system
Next, let's grant Fluentd access to the namespaces and pods in your Kubernetes cluster so that it may read, list, and monitor them. This is how the ClusterRole manifest should look:
apiVersion: rbac.authorization.k8s.io/v1betal kind: ClusterRole metadata: name: fluentd namespace: kube-system rules: - apiGroups: - " " resources: - pods - namespaces verbs: - get - list - watch
Finally, we must bind the Fluentd ServiceAccount to these permissions using the ClusterRoleBinding resource:
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1betal metadata: name: fluentd roleRef: kind: ClusterRole name: fluentd apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: fluentd namespace: kube-system
Using the —-delimiter, let's divide up these manifests and save them in the rbac.yml file before producing all resources at once:
kubectl create -f rbac.yml serviceaccount "fluentd" created clusterrole.rbac.authorization.k8s.io "fluentd" created clusterrole binding.rbac.authorization.k8s.io "fluentd" created
Step 2: Deploy a DaemonSet
We can utilize the Fluentd Kubernetes Daemonset example available in the Fluentd repository after making a few modifications.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: serviceAccount: fluentd serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:elasticsearch env: - name: FLUENT_ELASTICSEARCH_HOST value: "f505e785.qb0x.com" - name: FLUENT_ELASTICSEARCH_PORT value: "30216" - name: FLUENT_ELASTICSEARCH_SCHEME value: "https" - name: FLUENT_UID value: "0" # X-Pack Authentication # ===================== - name: FLUENT_ELASTICSEARCH_USER value: "abf54990f0a286dc5d76" - name: FLUENT_ELASTICSEARCH_PASSWORD value: "75c4bd6f7b" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
There are several parts of this configuration to which you should pay attention:
fluent/fluentd-kubernetes-daemonSet:elasticsearchDocker image is used by the DaemonSet to generate Fluentd output that is Elasticsearch.
/var/logand write pos files to the directory. In your DaemonSet manifest, set the
FLUENT_UIDenvironment variable to 0 to avoid permission issues.
Create a DaemonSet and store the manifest in the
kubectl create -f fluentd-elasticsearch.yml
If you run a single-node cluster with Minikube, as we did, the DaemonSet will create one Fluentd pod in the
kube-system namespace. You may check its logs using kubectl logs and find out its name with
kubectl get pods —namespace=kube-system.
With the given host and credentials, Fluentd will connect to Elasticsearch right away.
Let's view the logs that Fluentd has acquired by entering the Kibana dashboard. Under Management -> Index Patterns -> Create New Index Pattern, a new Logstash index produced by the Fluentd DaemonSet can be found. In order to transmit logs to Elasticsearch, Fluentd uses Logstash as an intermediary log shipper. After configuring a new index pattern, you may browse your app logs under the Discover tab.
You may find here a variety of logs generated by your Kubernetes apps and Kubernetes system components. A shared log document created by Fluentd contains the name of the stream that generated the log, the log message, as well as Kubernetes-specific information like the namespace, the Docker container ID, pod ID, and labels (see the example below).
log: INFO: == Kubernetes addon reconcile completed at 2018-09 11T09:31:39+0000 == stream: stdout docker.container id: 9b596c9195003246af0f71406f05ab4d339601dadc21304820 2992739fe9267e kubernetes.container_name: kube-addon-manager kubernetes.namespace name: kube-system kubernetes.pod_name: kube-addon-manager-mini kube kubernetes.pod id: f6d8ff9d-8a6e-11e8-9e55-0800270c281a kubernetes.labels.component: kube-addon-manager kubernetes.labels.version: v8.6
|Check Out: Kubernetes Vs Docker Swarm|
Typically, the Fluentd configuration file contains the following instructions:
Let's examine some of the most widely used Kubernetes Fluentd configuration options. The official GitHub repository's Kubernetes.conf file contains a complete example of the Kubernetes configuration.
<match **> @type stdout </match> <match fluent.**> @type null </match> <match docker> @type file path /var/log/fluent/docker.log time_slice_format %Y%m%d time_slice_wait 10m time_format %Y%m%dT%H%M%S%z compress gzip utc </match> <source> @type tail @id in_tail_container_logs path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true <% if is_v1 %> <parse> @type json time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> <% else %> format json time_format %Y-%m-%dT%H:%M:%S.%NZ <% end %> </source>
The three blocks of the above configuration are match directives. These directives filter logs by provider or name and determine the destination of the output using the @type variable for them.
As in the second match directive, @type null is used to ignore certain logs. In the third match directive, we filter Docker logs and write them to /var/log/fluent/docker.log
We have now reached the end of this blog. We hope that the comprehensive and practical explanation of how to implement Fluentd Kubernetes was useful.
However, if you think any crucial information is ignored? We request you to post your comments with your ideas.
Stay updated with our newsletter, packed with Tutorials, Interview Questions, How-to's, Tips & Tricks, Latest Trends & Updates, and more ➤ Straight to your inbox!
|Kubernetes Training||Aug 20 to Sep 04|
|Kubernetes Training||Aug 23 to Sep 07|
|Kubernetes Training||Aug 27 to Sep 11|
|Kubernetes Training||Aug 30 to Sep 14|
Kalla Saikumar is a technology expert and is currently working as a content associate at MindMajix. Write articles on multiple platforms such as ServiceNow, Business Analysis, Performance Testing, Mulesoft, Oracle Exadata, Azure, and other courses. And you can join him on LinkedIn.
Copyright © 2013 - 2022 MindMajix Technologies