tencent cloud

Feedback

TKE Event Log Analysis

Last updated: 2021-12-20 17:47:20

    Overview

    Situations in the cluster emerge one after another and are unpredictable, such as exceptional node status and pod restarts. If these situations cannot be perceived in the first place, users will miss the best time to deal with them. It's often too late to find out when the problem worsens and affects businesses.
    Event logs record comprehensive information about cluster status changes, helping users find and troubleshoot problems in the first place.

    Event Log Definition

    An event log is one of many resource objects in Kubernetes and is usually used to record status changes within a cluster, ranging from cluster node exceptions to pod startup and scheduling success. You can use the 'kubectl describe' command to view the event log information of resources.

    Event Log Fields

    img

    • Level (type): currently only the Normal and Warning levels are supported. If necessary, you can customize a level.
    • Resource type/object (involvedobject): objects involved in the event, such as Pod, Deployment, and Node.
    • Event source (source): component that reports the event, such as Scheduler and Kubelet.
    • Content (reason): brief description of the current event. Generally, an enumerated value is used. This field is used within the program.
    • Detailed description (message): detailed description of the current event.
    • Number of occurrences (count): number of times the event occurs

    Using Event Logs for Troubleshooting

    CLS provides a one-stop service for Kubernetes event logs, including collection, storage, search, and analysis capabilities. You only need to enable the cluster event log feature with a few clicks to obtain a visual event log analysis dashboard out of the box. With visual charts, you can easily solve most common OPS problems via the console.

    Prerequisites

    You have purchased the Tencent Kubernetes Engine (TKE) service and enabled the cluster event log feature. For more information, please see Operation Guide.

    Scenario 1: an exception occurred on a node, and you need to locate the cause

    1. Log in to the TKE console.
    2. On the left sidebar, choose Cluster OPS > Event Search.
    3. On the Event Search page, click Event Overview tab and enter the exception node name as the filter item.

    The query result is displayed. As shown in the following figure, an event is found in the Insufficient Node Disk Space area.

    Check the exception event trend and top exception events.

    As shown in the above figure, starting from 2020-11-25, the node 172.16.18.13 was exceptional due to insufficient disk space. Then Kubelet began to drain pods on the node to repossess the node's disk space.

    Scenario 2: a node triggered expansion, and you need to backtrack the expansion process to determine the cause

    For clusters with node pool auto scaling enabled, the Cluster Autoscaler (CA) component will automatically increase or decrease the number of nodes in the cluster based on the actual load. If the nodes in the cluster are automatically scaled out, users can trace the entire scaling process through event search.

    1. Log in to the TKE console.
    2. On the left sidebar, choose Cluster OPS > Event Search.
    3. On the Event Search page, click the Global Search tab, and enter the following search command:
      event.source.component : "cluster-autoscaler"
      
    4. In the Hidden Field on the left side, select event.reason, event.message, event.involvedObject.name, and event.involvedObject.name to display. Sort the query results in reverse order by Log Time. The result is as shown in the following figure.

    According to the event flow in the above figure, you can find that the node scaling occurred around 2020-11-25 20:35:45 and was triggered by three NGINX pods (nginx-5dbf784b68-tq8rd, nginx-5dbf784b68-fpvbx, and nginx-5dbf784b68-v9jv5). After three nodes were scaled out, the subsequent scaling was not triggered because the number of nodes in the node pool reached the upper limit.

    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support