When you are using NGINX Ingress and reducing the number of NGINX Ingress Controller replicas, the problem of "Connection Refused" may occur. In this case, RSs are unbound from CLB instances in batches, and TCP/UDP listeners stop forwarding existing connections.
Possible Causes
View the source code of NGINX Ingress and you can see that the workloads of NGINX Ingress Controller have no graceful shutdown capabilities. Therefore, a Pod directly exits after receiving the kill signal.
Solutions
If you use TKE's graceful service shutdown capabilities, when a Pod needs to be deleted, it can process the received requests, and inbound traffic is turned off while outbound traffic is still on. Outbound traffic will not be turned off until all existing requests are processed and the Pod is deleted. The Pod is deleted after the graceful shutdown period ends.
Troubleshooting
Note:
This is only effective in the direct access mode. Check whether your cluster supports direct access.
Step 1
Use an annotation to indicate the use of graceful shutdown in the ****-ingress-nginx-controller Service in the kube-system namespace.
The following is an example of using an annotation to indicate the use of graceful shutdown. For more information on Service annotations, see Service Annotation.
kind: Service
apiVersion: v1
metadata:
annotations:
service.cloud.tencent.com/direct-access: "true"## Enable CLB-to-Pod direct access
service.cloud.tencent.com/enable-grace-shutdown: "true"# Indicate the use of graceful shutdown
name: my-service
spec:
selector:
app: MyApp
Step 2
Add a sleep period before the wait-shutdown of the ****-ingress-nginx-controller Deployment in the kube-system namespace.
lifecycle:
preStop:
exec:
command:
- sleep# Add a sleep period
- 30s # Add a sleep period
- /wait-shutdown # Add a sleep period before this line
