Use Exit Code to Identify Pod Exceptions

Last updated: 2020-05-25 09:51:33

    This article describes how to use exit codes to troubleshoot pod issues.

    Querying Pod Exceptions

    Run the following command to query pod exceptions:

    kubectl describe pod <pod name>

    The returned result is as follows:

    Containers:
      kubedns:
        Container ID:  docker://5fb8adf9ee62afc6d3f6f3d9590041818750b392dff015d7091eaaf99cf1c945
        Image:         ccr.ccs.tencentyun.com/library/kubedns-amd64:1.14.4
        Image ID:      docker-pullable://ccr.ccs.tencentyun.com/library/kubedns-amd64@sha256:40790881bbe9ef4ae4ff7fe8b892498eecb7fe6dcc22661402f271e03f7de344
        Ports:         10053/UDP, 10053/TCP, 10055/TCP
        Host Ports:    0/UDP, 0/TCP, 0/TCP
        Args:
          --domain=cluster.local.
          --dns-port=10053
          --config-dir=/kube-dns-config
          --v=2
        State:          Running
          Started:      Tue, 27 Aug 2019 10:58:49 +0800
        Last State:     Terminated
          Reason:       Error
          Exit Code:    255
          Started:      Tue, 27 Aug 2019 10:40:42 +0800
          Finished:     Tue, 27 Aug 2019 10:58:27 +0800
        Ready:          True
        Restart Count:  1

    Exit Code is the status code of the last container exit. If it is not 0, then the container exited due to an exception. You can use the exit code to further troubleshoot the problem.

    Exit Codes

    • A valid exit code is between 0 and 255.
    • 0 means the container exited normally.
    • If the container exited due to an external signal, the exit code is between 129 and 255. For example, if the operating system sent kill -9 or ctrl+c as termination signals, the status is SIGKILL or SIGINT.
    • If the container exited due to an internal signal, the exit code is between 1 and 128. However, in some circumstances, the exit code might be between 129 and 255 too.
    • If the specified exit code has a value outside of the 0-255 range, such as exit(-1), it is automatically translated to a value in the 0-255 range.
      If the exit code is specified as code, it is translated as follows:
      • If the exit code is negative:
        256 - (|code| % 256)
      • If the exit code is positive:
        code % 256

    Typical Exit Codes

    • 137: the process was killed by SIGKILL. Possible reasons are:
      • Pod memory reached resources.limits, such as Out of Memory (OOM). Pod resource limits are implemented using Linux cgroup. If the memory of a pod reaches its limit, cgroup will force it to stop (similar to kill -9). If you use describe pod, you can see the value of Reason is OOMKilled.
      • The host does not have sufficient resources (OOM). The kernel selected processes to stop in order to release memory.

        If the process is stopped due to OOM, cgroup, or the host, you can find relevant records in system logs:
        Ubuntu system logs are stored in /var/log/syslog. CentOS system logs are stored in /var/log/messages. You can use journalctl -k to view system logs in both operating systems.

      • livenessProbe failed, which causes kubelet to stop the pod.
      • Pod stopped by a trojan process.
    • 1 and 255: common issues. Use container logs to further troubleshoot. For example, this could be the result of exit(1) or exit(-1). -1 is translated to 255.

    Standard Linux Interrupt Signals

    Linux programs send an exit code when they are interrupted by external signals. The value of the exit code is the value of the interrupt signal plus 128. For example, the value of SIGKILL is 9, so the program exit code is 9 + 128 = 137. For more standard interrupt signal, refer to the following table:

    /th> Value Action Description
    SIGHUP 1 Term Hangup detected on controlling terminal or death of controlling process
    SIGINT 2 Term Interrupt from keyboard
    SIGQUIT 3 Core Quit from keyboard
    SIGILL 4 Core Illegal Instruction
    SIGABRT 6 Core Abort signal from abort(3)
    SIGFPE 8 Core Floating-point exception
    SIGKILL 9 Term Kill signal
    SIGSEGV 11 Core Invalid memory reference
    SIGPIPE 13 Term Broken pipe: write to pipe with no readers; see pipe(7)
    SIGALRM 14 Term Timer signal from alarm(2)
    SIGTERM 15 Term Termination signal
    SIGUSR1 30,10,16 Term User-defined signal 1
    SIGUSR2 31,12,17 Term User-defined signal 2
    SIGCHLD 20,17,18 Ign Child stopped or terminated
    SIGCONT 19,18,25 Cont Continue if stopped
    SIGSTOP 17,19,23 Stop Stop process
    SIGTSTP 18,20,24 Stop Stop typed at terminal
    SIGTTIN 21,21,26 Stop Terminal input for background process
    SIGTTOU 22,22,27 Stop Terminal output for background process

    C/C++ Exit Codes

    /usr/include/sysexits.h provides standardized exit codes for C and C++. These codes are listed in the following table:

    Definition Exit Code Description
    #define EX_OK 0 Successful termination
    #define EX__BASE 64 Base value for error messages
    #define EX_USAGE 64 Command line usage error
    #define EX_DATAERR 65 Data format error
    #define EX_NOINPUT 66 Cannot open input
    #define EX_NOUSER 67 Addressee unknown
    #define EX_NOHOST 68 Host name unknown
    #define EX_UNAVAILABLE 69 Service unavailable
    #define EX_SOFTWARE 70 Internal software error
    #define EX_OSERR 71 System error (e.g., can't fork)
    #define EX_OSFILE 72 Critical OS file missing
    #define EX_CANTCREAT 73 Can't create (user) output file
    #define EX_IOERR 74 Input/output error
    #define EX_TEMPFAIL 75 Temp failure; user is invited to retry
    #define EX_PROTOCOL 76 Remote error in protocol
    #define EX_NOPERM 77 Permission denied
    #define EX_CONFIG 78 Configuration error
    #define EX__MAX 78 78 maximum listed value

    Reference

    For more information on exit codes, refer to Appendix E. Exit Codes With Special Meanings.

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help