Graceful Service Shutdown

Last updated: 2021-10-13 16:27:41

    Overview

    In scenarios where a Pod is connected directly at the access layer, when the backend performs a rolling update, or the backend Pod is deleted, if you delete the Pod directly from the CLB backend, unprocessed requests that have been received by it cannot be processed.
    Particularly, in persistent connection scenarios, such as meeting business, if the Pod of the workload is updated or deleted directly, the meeting will be interrupted.

    Use Cases

    • When a Pod quits gracefully during a workload update, the client will not perceive the jitters and errors generated during the update (if any).
    • When a Pod needs to be deleted, it can process the received requests, and inbound traffic is turned off while outbound traffic is still on. Outbound traffic will not be turned off until all existing requests are processed and the Pod is deleted.
    Note:

    This is only effective in the direct access mode. Please check whether your cluster supports direct access.

    Directions

    Step 1. Use an annotation to indicate the use of graceful shutdown

    The following is an example of using an annotation to indicate the use of graceful shutdown. For detailed Service annotations, please see Service Annotation.

    kind: Service
    apiVersion: v1
    metadata:
    annotations:
    service.cloud.tencent.com/direct-access: "true" ## Enable CLB-to-Pod direct access
    service.cloud.tencent.com/enable-grace-shutdown: "true" # Indicate the use of graceful shutdown
    name: my-service
    spec:
    selector:
    app: MyApp

    Step 2. Use preStop and terminationGracePeriodSeconds

    Step 2 involves using preStop and terminationGracePeriodSeconds in the workload that requires graceful shutdown.

    Container termination process

    The following describes the container termination process in a Kubernetes environment:

    1. The Pod is deleted, and the status is set to Terminating.
    2. kube-proxy updates the forwarding rule and removes the Pod from the endpoint list of the Service, and new traffic will no longer be forwarded to the Pod.
    3. If a preStop hook is configured for the Pod, it will be executed.
    4. kubelet will send a SIGTERM signal to each container in the Pod to ask the container processes to stop gracefully.
    5. Wait for the container processes to stop completely. If a process has not stopped completely after terminationGracePeriodSeconds (30s by default) elapses, a SIGKILL signal will be sent to forcibly stop it.
    6. All container processes are terminated, and the Pod resources are cleared.

    Specific steps

    1. Use preStop
      To implement graceful termination, you must process the SIGTERM signal in your business code. The main logic is to stop accepting new traffic, continue to process existing traffic, and quit after all connections are closed. For more information, please see Go by Example: Signals.
      If the SIGTERM signal is not processed in your business code, or if you cannot control the used third-party library or system to add the logic of graceful termination, you can also try configuring preStop for the Pod to implement such logic as shown below:

      apiVersion: v1
      kind: Pod
      metadata:
      name: lifecycle-demo
      spec:
      containers:
      - name: lifecycle-demo-container
      image: nginx
      lifecycle:
      preStop:
      exec:
      command:
      - /clean.sh
      ...
      For more information on how to configure `preStop`, please see Lifecycle.

      In certain extreme cases, new connections may still be forwarded within a short period of time after the Pod is deleted. This is because kubelet and kube-proxy watch that the Pod is deleted at the same time, and kubelet may have stopped the containers before kube-proxy syncs the rules. Normally, an application will no longer accept new connections after it receives SIGTERM, and it will only keep the existing connections for processing, which may cause some requests to fail at the moment when the Pod is deleted.

      In view of the above, you can use preStop to make the Pod sleep for a short while first and then start to stop the container processes after kube-proxy completes the rule sync as shown below:

      apiVersion: v1
      kind: Pod
      metadata:
      name: lifecycle-demo
      spec:
      containers:
      - name: lifecycle-demo-container
      image: nginx
      lifecycle:
      preStop:
      exec:
      command:
      - sleep
      - 5s
    2. Use terminationGracePeriodSeconds to adjust the termination grace period
      If you need a long termination grace period (preStop and the business process termination may take more than 30s in total), you can customize terminationGracePeriodSeconds as shown below based on the actual situation so as to avoid being stopped by SIGKILL prematurely:

      apiVersion: v1
      kind: Pod
      metadata:
      name: grace-demo
      spec:
      terminationGracePeriodSeconds: 60 # The termination grace period is 30s by default, and you can set a longer period.
      containers:
      - name: lifecycle-demo-container
      image: nginx
      lifecycle:
      preStop:
      exec:
      command:
      - sleep
      - 5s
      ...