Introduction

TKE's log collection feature allows you to collect logs in a cluster and send logs in specific paths of cluster services or nodes to Kafka, Elasticsearch, or Tencent Cloud Log Service (CLS). Log collection applies to users who need to store and analyze service logs in Kubernetes clusters.

Log collection must be manually enabled for each cluster. After log collection is enabled for a cluster, the log collection agent runs as a DaemonSet in the cluster, collects logs from the collection source, and sends the collected logs to the consumer based on the collection source and consumer configured using the log collection rules. Log collection supports the following operations:

• Collecting standard output logs of a container
• Collecting File logs of a pod
• Collecting File logs in specified node paths
• Configuring the consumer of logs

Prerequisites

• The cluster node has sufficient resources before log connection is enabled. Enabling log collection will occupy approximately 0.3 CPU cores and 250 MB memory of each cluster node by default.
  • CPU resources occupied: 0.3 CPU cores by default. You can increase this value if the log quantity is too large. We recommend that request be set to 1 core and limit be set to 2 cores.
  • Memory resources occupied: 250 MB by default. You can increase this value if the log quantity is too large. We recommend that request be set to 1 GB and limit be set to 1.5 GB.
  • Maximum log length: 512 KB for a log. A log will be truncated if its length exceeds the limit.
• To use log collection, ensure that nodes in a Kubernetes cluster can access the log consumer. Only Kubernetes clusters of version 1.10 or later support the following log collection features.

Concepts

• Log Collection Agent: the agent that TKE uses to collect logs. It is developed based on Fluentd and runs as a DaemonSet in a cluster.
• Log Collection Rules: you can specify the log collection source and the consumer to which the collected logs are sent.
  • The log collection agent monitors changes in the log collection rules, and changed rules take effect within 10 seconds.
  • Multiple DaemonSets will not be created if there are multiple log collection rules. However, too many log collection rules cause the log collection agent to occupy more resources.
• Log Source: this includes specified container logs and node path logs.
  • To collect standard output logs of services in a cluster, you can set the source to the specified container logs or service logs of all or several namespaces.
  • To collect logs in specific paths of cluster nodes, you can set the source to the node path logs. For example, to collect logs under paths in the format of /var/lib/docker/containers/<container-id>/<container-id>.json-log, you can specify the log collection path as /var/lib/docker/containers/*/*.json-log.
• Consumer end: after the log collection agent collects logs from the specified source, it will send these logs to the specified consumer.
  • The log collection service allows you to set the off-premises Elasticsearch, Kafka, Tencent Cloud's Ckafka service or Tencent Cloud Log Service (CLS) as the consumer of logs.
  • The log collection agent sends collected logs to the specified consumer in JSON format.

Directions

Collecting standard output logs of a container

The log collection feature allows you to collect standard output logs of a specified container in Kubernetes clusters. You can configure collection rules flexibly based on your needs.
The collected logs are sent to the specified consumer in JSON format with Kubernetes metadata, including the labels and annotations of the pod to which the container belongs.

Configuration method

1. Log in to the TKE console and click Log Collection in the left sidebar.
2. After selecting the region and cluster at the top of the log collection page, click Create, as shown below:
   
3. On the Create a log collection policy page, select Container standard output and configure the log source, as shown below:
   
   When you select container standard output as the collection type, the metadata below will be added for each log by default. log indicates the raw log information. This log source type allows you to select workloads of multiple namespaces at the same time.

   Field	Description
   docker.container_id	ID of the container to which logs belong
   kubernetes.annotations	Annotations of the pod to which logs belong
   kubernetes.container_name	Name of the container to which logs belong
   kubernetes.host	Host IP address of the pod to which logs belong
   kubernetes.labels	Labels of the pod to which logs belong
   kubernetes.namespace_name	Namespace of the pod to which logs belong
   kubernetes.pod_id	ID of the pod to which logs belong
   kubernetes.pod_name	Name of the pod to which logs belong
   log	Raw log information

4. Configure the consumer of logs. We recommend that Tencent Cloud CLS be set as the consumer of logs, as shown below:

5. Click Done to complete the creation.

Collecting file logs of a pod

The log collection feature also allows you to collect file logs of a specified pod in a cluster.
The collected logs are sent to the specified consumer in JSON format with Kubernetes metadata, including the labels and annotations of the pod to which the container belongs.

Currently, you can only collect log files stored in volumes. You must mount volumes such as emptyDir and hostpath when creating a workload and save the log files to the specified volume.

Configuration method

1. Log in to the TKE console and click Log Collection in the left sidebar.
2. After selecting the region and cluster at the top of the log collection page, click Create, as shown below:
   
3. Set the collection type to Container file path and configure the log source, as shown below:

   You can specify a path or use wildcards, for example, /var/log/nginx.log or /var/lib/docker/containers/*/*.log to collect log files in the corresponding paths of the pod.

When you select container file path as the collection type, the metadata below is added for each log by default. message indicates the raw log information. This log source type does not support selecting workloads of multiple namespaces.

Field	Description
docker.container_id	ID of the container to which logs belong
kubernetes.annotations	Annotations of the pod to which logs belong
kubernetes.container_name	Name of the container to which logs belong
kubernetes.host	Host IP address of the pod to which logs belong
kubernetes.labels	Labels of the pod to which logs belong
kubernetes.namespace_name	Namespace of the pod to which logs belong
kubernetes.pod_id	ID of the pod to which logs belong
kubernetes.pod_name	Name of the pod to which logs belong
file	Source log file
message	Raw log information

4. Configure the consumer of logs. We recommend that Tencent Cloud CLS be set as the consumer of logs, as shown below:

5. Click Done to complete the creation.

Collecting file logs in specified node paths

The log collection feature allows you to collect logs in specified node paths of all nodes in a cluster. You can configure the required paths flexibly based on your needs. The log collection agent collects file logs in paths that meet the specified path rules of all nodes in the cluster.
The collected logs are sent to the specified consumer in JSON format with specified metadata, including the source file path and custom metadata.

Configuration method

1. Log in to the TKE console and click Log Collection in the left sidebar.
2. After selecting the region and cluster at the top of the log collection page, click Create, as shown below:
   
3. On the Create a log collection policy page, select Node file path, as shown below:

   You can specify a path or use wildcards, for example, /var/log/nginx.log or /var/lib/docker/containers/*/*.log to collect log files in the corresponding paths of all nodes in the cluster.

You can add custom metadata as needed and attach metadata specified in key-value format to the collected logs as their metadata tags.
Attached metadata will be added to the logs in JSON format, as shown below:

For example, Without specified metadata attached, the collected logs appear as below:

With specified metadata attached, the collected logs appear as below:

Compared with logs without specified metadata attached, JSON logs with metadata attached have an additional key service.
Log metadata is defined as follows:

Field	Description
path	Source file of logs
message	Log information
Custom key	Custom value

4. Configure the consumer of logs. We recommend that Tencent Cloud CLS be set as the consumer of logs, as shown below:

5. Click Done to complete the creation.

Configuring consumer of logs

The log collection feature allows you to set off-premises Kafka pods, topics specified by Tencent Cloud Ckafka pods, or log topics specified by Tencent Cloud Log Service (CLS) as the consumer of logs. The log collection agent will send the collected logs to the topic specified by Kafka or the log topic specified by CLS.

Configuring Kafka as the consumer of logs

Only Kafka pods without access authentication are supported. All nodes in the cluster must be able to access the specified Kafka topic.
If you use the Ckafka service provided by Tencent Cloud, select a Ckafka pod. Otherwise, enter the Kafka access address and topic, as shown below:

Configuring CLS as the consumer of logs

Currently, CLS only supports log collection and reporting for intra-region container clusters.

1. Because TKE's logs can be independently collected, you don't have to enable LogListener to create a logset, as shown below:
   For more information on how to create a log set, see Creating a logset and a log topic.
   
2. Enable the log topic's Log Index, as shown below:
   

Configuring Elasticsearch as the consumer of logs

Only Elasticsearch services without access authentication are supported. All nodes in the cluster must be able to access the specified Elasticsearch service.
Enter the Elasticsearch service's access address and storage index, as shown below: