Using Environment Variables to Configure Log Collection

Last updated: 2021-07-14 15:03:54

    Overview

    In EKS, you can use environment variables to configure log collection, and collect logs by line without parsing, or use custom resource definitions (CRD) to configure log collection.

    This document describes how to use the environment variables to configure the log collection feature of the EKS cluster, and to send the logs of services within the cluster to Cloud Log Service (CLS) or external Kafka. This feature is suitable for users who need to store and analyze service logs in EKS clusters.

    To use the EKS log collection feature, you need to manually enable it for each elastic cluster when creating a workload. You can enable this feature by performing the following operations:

    Notes

    After the EKS log collection feature is enabled, the log collection agent will send the collected logs in JSON format to the consumer end that you have specified based on your configuration of the collection path and log consumer end. The details of the collection path and consumer end are as follows:

    • Consumer end: the log collection service allows you to set Kafka or CLS as the log consumer end.
    • Collection path: the path of the logs to collect. The collection path supports collection standard output (stdout) and absolute paths. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.

    Prerequisites

    Directions

    Configuring log collection via the console

    The EKS log collection feature collects the log information and outputs it to the specified consumer end in JSON format with Kubernetes metadata attached, including the label of the Pod to which the container belongs, annotation, etc. The specific directions are as follows:

    1. Log in to the TKE console, and click Elastic Cluster in the left sidebar.

    2. On the "Elastic Cluster" page, click the ID of the desired cluster for log collection to enter the cluster management page.

    3. Select a workload type in Workload on the left to go to the corresponding page, and then click Create.

    4. In Containers in the Pod section, select Advanced Settings, and check Activate to enable log collection, as shown in the figure below:

    5. Refer to the following information to configure the log consumer end. You can choose CLS or Kafka as the log consume end.

      1. Select CLS as the Consumer End, and select the Log set and Log topic, as shown below:

        If there is no suitable log set, you can create a logset and a log topic.
        Note

        CLS currently only supports log collection and reporting for intra-region container clusters.

      2. Enable the log index for the selected log topic. Index configuration is required for CLS log search and analysis. If it is not enabled, you cannot view the logs. For how to configure index, see Configuring Index.
        You can go to the CLS console > Log Topic page, select a log topic name, and enable the index in the Index Configuration tab, as shown in the figure below:
    6. Select Role or Key to authorize.

      Note:

      • You can only select the same authorization method for the containers in the same Pod. The last modification shall prevail. For example, if you select key authorization for the first container and role authorization for the second container, finally both containers will adopt role authorization.
      • You can only select the same role to authorize for the containers in the same Pod.
    • Select a role that can access CLS, as shown in the figure below:
    • If there is no suitable role, you can create one as follows:
      1. Log in to the CAM console, and select [Roles](https://console.cloud.tencent.com/cam/role) in the left sidebar.
      2. On the Roles page, click Create Role.
      3. In the “Select role entity” window that appears, select Tencent Cloud Product Service to go to the Create Custom Role page.
      4. On the Enter role entity info tab, select Cloud Virtual Machine (cvm) and click Next.
        Note

        You must select Cloud Virtual Machine (cvm) rather than TKE as the role entity, otherwise, you cannot complete the authorization process.

      5. On the Configure role policy tab, select QcloudCLSAccessForApiGateWayRole policy and click Next.
      6. On the Review tab, enter the role name to review the role information, and then click Done. For more information, see Creating a Role.
    1. Configure the collection path, as shown in the figure below:

      At this point, you have configured the log collection feature. You can set other configurations of the workload as needed.

    Configuring log collection via yaml

    This document provides three collection methods for your choice: collecting logs to Kafka, collecting logs to CLS via a secret and collecting logs to CLS via a role.

    Note:

    If both key and role authorization are configured in yaml, Pod actually uses role authorization.

    Enable log collection by adding environment variables.

    apiVersion: apps/v1beta2
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "1"
    labels:
        k8s-app: kafka
        qcloud-app: kafka
      name: kafka
      namespace: default
    spec:
      replicas: 1
      selector:
        matchLabels:
          k8s-app: kafka
          qcloud-app: kafka
     template:
    metadata:
          annotations:
            eks.tke.cloud.tencent.com/cpu: "0.25"
            eks.tke.cloud.tencent.com/mem: "0.5Gi"
    labels:
            k8s-app: kafka
            qcloud-app: kafka
        spec:
          containers:
          - env:
            - name: EKS_LOGS_OUTPUT_TYPE
              value: kafka
            - name: EKS_LOGS_KAFKA_BROKERS
              value: 10.0.16.42:9092
            - name: EKS_LOGS_KAFKA_TOPIC
              value: eks
            - name: EKS_LOGS_KAFKA_MESSAGE_KEY # Optional
              valueFrom:
                 fieldRef:
                   fieldPath: metadata.name
            - name: EKS_LOGS_METADATA_ON
              value: "true"
            - name: EKS_LOGS_LOG_PATHS
              value: stdout,/tmp/busy*.log
            image: busybox:latest
            command: ["/bin/sh"]
            args: ["-c", "while true; do echo hello world; date; echo hello >> /tmp/busy.log; sleep 1; done"]
            imagePullPolicy: Always
            name: while
            resources:
              requests:
                cpu: 250m
                memory: 512Mi
    

    Field description:

    Field Name Description
    EKS_LOGS_OUTPUT_TYPE The consumer end can be kafka or CLS. The key indicates whether log collection is enabled.
    EKS_LOGS_LOG_PATHS Log path. It supports stdout (collection standard output) and absolute paths. It also supports wildcard (*). If multiple paths are specified, separate them with “,”.
    EKS_LOGS_METADATA_ON Valid values: `true` or `false`. Default value: `true`.
    EKS_LOGS_KAFKA_TOPIC Log topic
    EKS_LOGS_KAFKA_BROKERS kafka brokers: ip1:port1, ip1:port2, and ip2:port2 formats, separated by “,”. Use this environmental variable for external application. EKS_LOGS_KAFKA_HOST will no longer be visible to external users.
    EKS_LOGS_KAFKA_MESSAGE_KEY Optional. A key can be specified to deliver the log to the specified partition.
  • If the delivery by key is not enabled, logs will be randomly delivered to different partitions.
  • If the delivery by key is enabled, logs with the same key will be delivered to the same partition.
  • Note: here the key gets the variable value from the field of the Pod. The above examples all take the Pod name as an example, and it also supports namespace, PodIP, etc. For more information, see Kubernetes Documentation.

    Updating the log collection

    You can update log collection via the console and yaml. Please refer to the following directions:

    1. Log in to the TKE console, and click Elastic Cluster in the left sidebar.
    2. Select the ID of the desired cluster for log collection configuration to enter the cluster management page.
    3. Choose Workload on the left, find the row of the desired workload for which you want to update log collection, and click Update Pod Configuration > Advanced Settings on the right to modify configurations, as shown in the figure below:
    4. Click Done.