Using CRD to Configure Log Collection via YAML

Prev Next

search by keyword

Recent Pages

Documentation

Download PDF

Using CRD to Configure Log Collection via YAML

Last updated: 2022-03-23 10:54:16

Download PDF

Overview

Besides using the TKE console, you can also configure the log collection settings by using the Custom Resource Definitions (CRD). CRD supports the collection of container standard outputs, container files, and host files. It also supports multiple log collection formats, and supports shipping logs to different consumers such as CLS and CKafka.

Prerequisites

You have enabled Log Collection in Feature Management in the TKE console. For more information, see Enabling Log Collection.

Creating CRD to Ship Logs to CLS

To create a collection configuration, you only need to define the LogConfig CRD. The log-agent modifies the corresponding CLS log topics based on changes to the LogConfig CRD and set the bound server group. The CRD format is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                              ## Default value
metadata:
 name: test                                ## CRD resource name, unique in the cluster
spec:
 clsDetail:
   # Note: You cannot modify the topic after it is specified.
   # If the log topic is created automatically, the names of logset and topic need to be specified at the same time.
   logsetName: test                        ## CLS logset name. Logset for the name will be created automatically if there is not any. If there is the logset, log topic will be created under it.
   topicName: test                         ## CLS log topic name. Log topic for the name will be created automatically if there is not any.
    
   # Select an existing log topic
   topicId: xxxxxx-xx-xx-xx-xxxxxxxx       ## CLS log topic ID. The log topic needs to be created in CLS in advance and should not be occupied by other collection configurations.
   logType: minimalist_log                 ## Log collection format. json_log: json format. delimiter_log: separator-based format. minimalist_log: full text in a single line. multiline_log: full text in multi lines. fullregex_log: full regex format.
   extractRule:                            ## Extraction and filtering rule
     ...
 inputDetail:
   type: container_stdout                  ## Log collection type, including container_stdout (container standard output), container_file (container file), and host_file (host file)
   
   containerStdout:                        ## Container standard output
     namespace: default                    ## The Kubernetes namespace of the container to be collected. If this parameter is not specified, it indicates all namespaces.
     allContainers: false                  ## Whether to collect the standard output of all containers in the specified namespace
     container: xxx                        ## Name of the container of which the logs will be collected. If the name is empty, it indicates the log names of all matching containers will be collected.
     includeLabels:                         ## Only Pods that contain the specified labels will be collected.
       k8s-app: xxx                        ## Only the logs generated by Pods with the configuration of "k8s-app=xxx" in the Pod labels will be collected. This parameter cannot be specified at the same time as workloads and allContainers=true.
     workloads:                            ## Kubernetes workload to which the container Pod to be collected belongs
     - namespace: prod                     ## Workload namespace
       name: sample-app                    ## Workload name
       kind: deployment                    ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
       container: xxx                      ## Name of the container to collect. If this parameter is not specified, it indicates all containers in the workload Pod will be collected.

   containerFile:                          ## File in the container
     namespace: default                    ## The Kubernetes namespace of the container to be collected. A namespace must be specified.
     container: xxx                        ## The name of container of which the logs will be collected. The * indicates the log names of all matching containers will be collected.
    includeLabels:                         ## Only Pods that contain the specified labels will be collected.
       k8s-app: xxx                        ## Only the logs generated by Pods with the configuration of "k8s-app=xxx" in the Pod labels are collected. This parameter cannot be specified at the same time as workload.
     workload:                             ## Kubernetes workload to which the container Pod to be collected belongs
       name: sample-app                    ## Workload name                  
       kind: deployment                    ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
     logPath: /opt/logs                    ## Log folder. Wildcards are not supported.
     filePattern: app_*.log                ## Log file name. It supports the wildcards "*" and "?". "*" matches multiple random characters, and "?" matches a single random character.
    
   hostFile:                               ## Host file
     logPath: /opt/logs                    ## Log folder. Wildcard is supported.
     filePattern: app_*.log                ## Log file name. It supports the wildcards "*" and "?". "*" matches multiple random characters, and "?" matches a single random character.
     customLablels:
       k1: v1

Note

If the collection type is selected as "Container File Path", the corresponding path cannot be a soft link. Otherwise, the actual path of the soft link will not exist in the collector's container, resulting in log collection failure.

A log with full text in a single line means a line is a full log. When CLS collects logs, it uses the line break \n to mark the end of a log. For easier structural management, a default key value __CONTENT__ is given to each log, but the log data itself will no longer be structured, nor will the log field be extracted. The time attribute of a log is determined by the collection time. For more information, see Full Text in a Single Line.

Assume that the raw data of a log is as follows:

Tue Jan 22 12:08:15 CST 2019 Installed: libjpeg-turbo-static-1.2.90-6.el7.x86_64

A sample of LogConfig configuration is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 clsDetail:
   topicId: xxxxxx-xx-xx-xx-xxxxxxxx
   # Single-line log
   logType: minimalist_log

The data collected to CLS is as follows:

__CONTENT__:Tue Jan 22 12:08:15 CST 2019 Installed: libjpeg-turbo-static-1.2.90-6.el7.x86_64

A log with full text in multi lines means that a full log may occupy multiple lines (such as Java stacktrace). In this format, the line break \n cannot be used as the end mark of a log. To help the CLS system distinguish among the logs, "First Line Regular Expression" is used for matching. When a log in a line matches the preset regular expression, it is considered the beginning of a log, and the next matching line will be the end mark of the log. In this format, a default key value __CONTENT__ is also set. However, the log data itself is no longer structured, and the log fields are not extracted. The Time log attribute depends on the time the log is collected. For more information, see Full Text in Multi Lines.

Assume that the raw data of a multi-line log is:

2019-12-15 17:13:06,043 [main] ERROR com.test.logging.FooFactory:
java.lang.NullPointerException
     at com.test.logging.FooFactory.createFoo(FooFactory.java:15)
     at com.test.logging.FooFactoryTest.test(FooFactoryTest.java:11)

A sample of LogConfig is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec: 
  clsDetail: 
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Multi-line log
    logType: multiline_log
    extractRule: 
      # Only a line that starts with a date time is considered the beginning of a new log. Otherwise, add the line break \n to the end of the current log.
      beginningRegex: \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s.+

The data collected to CLS is as follows:

__CONTENT__:2019-12-15 17:13:06,043 [main] ERROR com.test.logging.FooFactory:\njava.lang.NullPointerException\n    at com.test.logging.FooFactory.createFoo(FooFactory.java:15)\n    at com.test.logging.FooFactoryTest.test(FooFactoryTest.java:11)

Full Regex is often used to process structured logs. It parses a full log by extracting multiple key-value pairs based on a regex. For more information, see Collecting Full RegEx Logs.
Assume that the raw data of a log is as follows:

10.135.46.111 - - [22/Jan/2019:19:19:30 +0800] "GET /my/course/1 HTTP/1.1" 127.0.0.1 200 782 9703 "http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0"  0.354 0.354

A sample of LogConfig is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec: 
  clsDetail: 
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Full Regex
    logType: fullregex_log
    extractRule: 
      # Regular expression, in which the corresponding values will be extracted based on the () capture groups
      logRegex: (\S+)[^[]+([[^:]+:\d+:\d+:\d+\s\S+)\s"(\w+)\s(\S+)\s([^"]+)"\s(\S+)\s(\d+)\s(\d+)\s(\d+)\s"([^"]+)"\s"([^"]+)"\s+(\S+)\s(\S+).
      beginningRegex: (\S+)[^[]+([[^:]+:\d+:\d+:\d+\s\S+)\s"(\w+)\s(\S+)\s([^"]+)"\s(\S+)\s(\d+)\s(\d+)\s(\d+)\s"([^"]+)"\s"([^"]+)"\s+(\S+)\s(\S+).
      # List of extracted keys, which are in one-to-one correspondence with the extracted values
      keys:   ['remote_addr','time_local','request_method','request_url','http_protocol','http_host','status','request_length','body_bytes_sent','http_referer','http_user_agent','request_time','upstream_response_time']

The data collected to CLS is as follows:

body_bytes_sent: 9703
http_host: 127.0.0.1 
http_protocol: HTTP/1.1
http_referer: http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum
http_user_agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
remote_addr: 10.135.46.111 
request_length: 782
request_method: GET
request_time: 0.354
request_url: /my/course/1
status: 200
time_local: [22/Jan/2019:19:19:30 +0800]
upstream_response_time: 0.354

The multi-line - full regular expression mode is a log parsing mode where multiple key-value pairs can be extracted from a complete piece of log data that spans multiple lines in a log text file (such as Java program logs) based on a regular expression. If you don't need to extract key-value pairs, please configure it as instructed in full text in multi lines. For more information, see Full Text in Multi Lines.

Assume that the raw data of a log is as follows:

[2018-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
   at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
   at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
   at TestPrintStackTrace.main(TestPrintStackTrace.java:16)

A sample of LogConfig is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec: 
  clsDetail: 
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
        # Multiple lines - full regex
        logType: multiline_fullregex_log
        extractRule: 
      # The first-line full regular expression: only a line that starts with a date time is considered the beginning of a new log. Otherwise, add the line break \n to the end of the current log.
            beginningRegex: [\d+-\d+-\w+:\d+:\d+,\d+]\s[\w+]\s.
      # Regular expression, in which the corresponding values will be extracted based on the () capture groups
      logRegex: [(\d+-\d+-\w+:\d+:\d+,\d+)]\s[(\w+)]\s(.)
            # List of extracted keys, which are in one-to-one correspondence with the extracted values
            keys: 
      - time 
      - level 
      - msg

Based on the extracted key, the data collected to CLS is as follows:

time: 2018-10-01T10:30:01,000<br>level: INFO
msg: java.lang.Exception: exception happened
   at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
   at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
   at TestPrintStackTrace.main(TestPrintStackTrace.java:16)

A JSON log automatically extracts the key at the first layer as the field name and the value at the first layer as the field value to implement structured processing of the entire log. Each complete log ends with a line break \n. For more information, see JSON Format.

Assume the raw data of a JSON log is as follows:

{"remote_ip":"10.135.46.111","time_local":"22/Jan/2019:19:19:34 +0800","body_sent":23,"responsetime":0.232,"upstreamtime":"0.232","upstreamhost":"unix:/tmp/php-cgi.sock","http_host":"127.0.0.1","method":"POST","url":"/event/dispatch","request":"POST /event/dispatch HTTP/1.1","xff":"-","referer":"http://127.0.0.1/my/course/4","agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0","response_code":"200"}

A sample of LogConfig is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
  clsDetail:
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # JSON log
    logType: json_log

The data collected to CLS is as follows:

agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
body_sent: 23
http_host: 127.0.0.1
method: POST
referer: http://127.0.0.1/my/course/4
remote_ip: 10.135.46.111
request: POST /event/dispatch HTTP/1.1
response_code: 200
responsetime: 0.232
time_local: 22/Jan/2019:19:19:34 +0800
upstreamhost: unix:/tmp/php-cgi.sock
upstreamtime: 0.232
url: /event/dispatch
xff: -

In a separator log, the entire log data can be structured according to the specified separator, and each complete log ends with a line break \n. When CLS processes separator logs, you need to define a unique key for each separate field. For more information, see Separator Format.

Assume the raw log is as follows:

10.20.20.10 ::: [Tue Jan 22 14:49:45 CST 2019 +0800] ::: GET /online/sample HTTP/1.1 ::: 127.0.0.1 ::: 200 ::: 647 ::: 35 ::: http://127.0.0.1/

A sample of LogConfig is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec: 
  clsDetail: 
    topicId: xxxxxx-xx-xx-xx-xxxxxxxx
    # Separator log
    logType: delimiter_log
    extractRule: 
      # Separator
      delimiter: ':::'
      # List of extracted keys, which are in one-to-one correspondence to the separated fields
      keys: ['IP','time','request','host','status','length','bytes','referer']

The data collected to CLS is as follows:

IP: 10.20.20.10 
bytes: 35
host: 127.0.0.1 
length: 647
referer: http://127.0.0.1/
request: GET /online/sample HTTP/1.1
status: 200
time: [Tue Jan 22 14:49:45 CST 2019 +0800]

Creating CRD to Ship Logs to CKafka

You can configure CRD to ship the logs of Pods in TKE to external Kafka service or Tencent Cloud CKafka. You need to define the log source and consumer according to the following configurations.
The specific configuration of CRD is as follows:

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig                          ## Default value
metadata:
 name: test                                ## CRD resource name, unique in the cluster
spec:
 kafkaDetail:
   kafkaType:       ## (Required) Kafka service type. "ckafka" indicates Tencent Cloud CKafka and "" indicates external Kafka services.
     type: string
   instanceId:     ## ckafka instance ID.
     type: string
   brokers: xxxxxx       ## (Required) The broker address. Generally, it is domain name:port. If there are more than one address, separate them with ",".
   topic: xxxxxx         ## (Required) Topic name.        
   messageKey:           ## (Optional) You can specify the Pod field as the key to upload to the specified partition.
     valueFrom:
       fieldRef:
         fieldPath: metadata.name   
   timestampKey:            ## The key of timestamp. Default value is @timestamp.
   timestampFormat:       ## The format of timestamp. Default value is double.
 inputDetail:
   type: container_stdout                  ## Log collection type, including container_stdout (container standard output) and container_file (container file).
    containerStdout:                        ## Container standard output
     namespace: default                    ## The Kubernetes namespace of the container to be collected. If this parameter is not specified, it indicates all namespaces.
     allContainers: false                  ## Whether to collect the standard output of all containers in the specified namespace
     container: xxx                        ## Name of the container to be collected. This item can be left empty.
     includeLabels:                         ## Only Pods that contain the specified labels will be collected.
       k8s-app: xxx                        ## Only the logs generated by Pods with the configuration of "k8s-app=xxx" in the Pod labels will be collected. This parameter cannot be specified at the same time as workloads and allContainers=true.
     workloads:                            ## Kubernetes workload to which the container Pod to be collected belongs
     - namespace: prod                     ## Workload namespace
       name: sample-app                    ## Workload name
       kind: deployment                    ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
       container: xxx                      ## Name of the container to be collected. If this item is left empty, it indicates all containers in the workload Pod will be collected.
    containerFile:                          ## File in the container
     namespace: default                    ## The Kubernetes namespace of the container to be collected. A namespace must be specified.
     container: xxx                        ## Name of the container to be collected. You can enter a * for this item.
     includeLabels:                         ## Only Pods that contain the specified labels will be collected.
       k8s-app: xxx                        ## Only the logs generated by Pods with the configuration of "k8s-app=xxx" in the Pod labels are collected. This parameter cannot be specified at the same time as workload.
     workload:                             ## Kubernetes workload to which the container Pod to be collected belongs
       name: sample-app                    ## Workload name                  
       kind: deployment                    ## Workload type. Supported values include deployment, daemonset, statefulset, job, and cronjob.
     logPath: /opt/logs                    ## Log folder. Wildcards are not supported.
     filePattern: app_*.log                ## Log file name. It supports the wildcards "*" and "?". "*" matches multiple random characters, and "?" matches a single random character.
    
     ...

Log Collection Types

Container standard output

Sample 1: collecting the standard output of all containers in the default namespace

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 inputDetail:
   type: container_stdout
   containerStdout:
     namespace: default
     allContainers: true
...

Sample 2: collecting the container standard output in the Pod that belongs to ingress-gateway deployment in the production namespace

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 inputDetail:
   type: container_stdout
   containerStdout:
     allContainers: false
     workloads:
     - namespace: production
       name: ingress-gateway
       kind: deployment
 ...

Sample 3: collecting the container standard output in the Pod whose Pod labels contain “k8s-app=nginx” under the production namespace

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 inputDetail:
   type: container_stdout
   containerStdout:
     namespace: production
     allContainers: false
     includeLabels:
       k8s-app: nginx
 ...

Container file

Sample 1: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod that belongs to ingress-gateway deployment under the production namespace

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 topicId: xxxxxx-xx-xx-xx-xxxxxxxx
 inputDetail:
   type: container_file
   containerFile:
     namespace: production
     workload:
       name: ingress-gateway
       type: deployment
     container: nginx
     logPath: /data/nginx/log
     filePattern: access.log
 ...

Sample 2: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod whose pod labels contain “k8s-app=ingress-gateway” under the production namespace

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 inputDetail:
   type: container_file
   containerFile:
     namespace: production
     includeLabels:
       k8s-app: ingress-gateway
     container: nginx
     logPath: /data/nginx/log
     filePattern: access.log
 ...

Host file

Sample: collect all `.log` files in the host path `/data/`

apiVersion: cls.cloud.tencent.com/v1
kind: LogConfig
spec:
 inputDetail:
   type: host_file
   hostFile:
     logPath: /data
     filePattern: *.log
 ...

Metadata

For container standard output (container_stdout) and container files (container_file), in addition to the raw log content, the container metadata (for example, the ID of the container that generated the logs) also needs to be carried and reported to CLS. In this way, when viewing logs, users can trace the log source or search based on the container identifier or characteristics (such as container name and labels).
The following table lists the metadata:

Field Name	Description
cluster_id	The ID of the cluster to which logs belong
container_name	The name of the container to which logs belong
container_id	ID of the container to which logs belong
image_name	The image name IP of the container to which logs belong
namespace	The namespace of the Pod to which the logs belong
pod_uid	The UID of the Pod to which the logs belong
pod_name	The name of the Pod to which the logs belong
pod_ip	The IP of the Pod to which the logs belong
pod_lable_{label name}	The labels of the Pod to which the logs belong (for example, if a Pod has two labels: app=nginx and env=prod, the reported log will have two metadata entries attached: pod_label_app:nginx and pod_label_env:prod).

Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support

tencent cloud

Tencent Cloud Partner Network

Become a Partner

International Partner Academy

Recent Pages

Using CRD to Configure Log Collection via YAML

Overview

Prerequisites

Creating CRD to Ship Logs to CLS

Configuring the CLS log parsing format

Creating CRD to Ship Logs to CKafka

Log Collection Types

Container standard output

Sample 1: collecting the standard output of all containers in the default namespace

Sample 2: collecting the container standard output in the Pod that belongs to ingress-gateway deployment in the production namespace

Sample 3: collecting the container standard output in the Pod whose Pod labels contain “k8s-app=nginx” under the production namespace

Container file

Sample 1: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod that belongs to ingress-gateway deployment under the production namespace

Sample 2: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod whose pod labels contain “k8s-app=ingress-gateway” under the production namespace

Host file

Sample: collect all `.log` files in the host path `/data/`

Metadata

Was this page helpful?

Was this page helpful?

About Tencent Cloud

Help & Support

User Center

Resources

tencent cloud

Become a Partner

Tencent Cloud Partner Network

Become a Partner

Training & Support

International Partner Academy

Sign Up

Log in

Recent Pages

Using CRD to Configure Log Collection via YAML

Overview

Prerequisites

Creating CRD to Ship Logs to CLS

Configuring the CLS log parsing format

Creating CRD to Ship Logs to CKafka

Log Collection Types

Container standard output

Sample 1: collecting the standard output of all containers in the default namespace

Sample 2: collecting the container standard output in the Pod that belongs to ingress-gateway deployment in the production namespace

Sample 3: collecting the container standard output in the Pod whose Pod labels contain “k8s-app=nginx” under the production namespace

Container file

Sample 1: collecting the access.log file in the /data/nginx/log/ path in the nginx container in the Pod that belongs to ingress-gateway deployment under the production namespace

Sample 2: collecting the access.log file in the /data/nginx/log/ path in the nginx container in the Pod whose pod labels contain “k8s-app=ingress-gateway” under the production namespace

Host file

Sample: collect all .log files in the host path /data/

Metadata

Was this page helpful?

Was this page helpful?

About Tencent Cloud

Help & Support

User Center

Resources

Sample 1: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod that belongs to ingress-gateway deployment under the production namespace

Sample 2: collecting the `access.log` file in the `/data/nginx/log/` path in the nginx container in the Pod whose pod labels contain “k8s-app=ingress-gateway” under the production namespace

Sample: collect all `.log` files in the host path `/data/`