The cluster auditing and event storage features of TKE are configured with rich visual charts to display audit logs and cluster events in multiple dimensions. Their operations are simple, and most common cluster Ops use cases are covered, making it easy for you to find and locate problems, improve the Ops efficiency, and maximize the value of audit and event data.
This document describes how to use audit and event dashboards to quickly locate cluster problems for several use cases.
10001****7138account deleted the
2020-11-30T03:37:13. For more information on the account, select CAM > User List.
10001****7138account cordoned the
tke-kube-state-metricsuser has much more access requests than others. The operation type distribution trend shows that most of the operations are LIST operations, and the status code distribution trend shows that most of the status codes are 403. The business logs show that the
tke-kube-state-metricsadd-on kept requesting API server retries due to the RBAC authentication issue, resulting in a sharp increase in API server access requests. Below is a sample log:
E1130 06:19:37.368981 1 reflector.go:156] email@example.com/tools/cache/reflector.go:108: Failed to list *v1.VolumeAttachment: volumeattachments.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:tke-kube-state-metrics" cannot list resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
172.16.18.13node was abnormal due to insufficient disk space, after which kubelet started to try evicting Pods on the node to repossess the disk space.
The cluster auto-scaler (CA) add-on automatically increases or decreases the number of nodes in the cluster according to the load condition when node pool elastic scaling is enabled. If a node in the cluster is automatically scaled, you can backtrack the whole scaling process through event search.
Log in to the TKE console.
On the left sidebar, select Cluster Ops > Event Search.
Select the Global Search tab and enter the following search command in the search box:
event.source.component : "cluster-autoscaler"
event.involvedObject.name from the Hidden Fields on the left for display. Click Search and Analysis and view the results.
Sort the search results by Log Time in reverse order as shown below:
As shown above, the event shows that the node scale-out occurred at
2020-11-25 20:35:45, which was triggered by three Nginx Pods (
nginx-5dbf784b68-v9jv5). As a result, three nodes were added, and further scale-out was not triggered as the maximum number of nodes was reached in the node pool.