tencent cloud

Feedback

Abnormal TaskManager Pod Exit

Last updated: 2023-11-07 16:57:54

    Overview

    TaskManagers of a Flink Job run in a Kubernetes Pod. When the Pod stops, the Pod exit event is generated, and you can determine whether the Pod is in exception based on the return code, status, and other information.
    Note
    The same Pod may be re-built several times by Kubernetes due to exceptions, so you may receive an identical event several times.

    Trigger conditions

    The system monitors the exit of the TaskManager Pod in real time, and determines whether an exit is caused by SIGTERM based on the exit code (the normal exit code is 143). An exit code other than 143 indicates that the exit is not initiated by the JobManager, but is caused by TaskManager errors, and this case is determined as an abnormal Pod exit event.

    Alarm configuration

    You can configure an alarm policy as instructed in Configuring Event Alarms (Events) for this event to receive trigger and clearing notifications in real time.

    Suggestions

    Status Code
    Possible Cause
    Solution
    137
    The memory occupied by the job exceeded the memory quota of the Pod, and the Pod was killed due to OOM.
    Increase the operator parallelism and the Task Manager spec (CUs) as instructed in Configuring Job Resources.
    -1
    This is the code of the basic policy, indicating that the Pod has exited, but no exit code is returned due to system errors or other reasons.
    Submit a ticket to contact the technicians for help.
    0
    During the startup process, the Pod cannot assign IPs in the associated subnet (no IPs available, for example), resulting in startup failure and exit.
    Check whether available IPs are sufficient in the VPC subnet associated with the cluster. If yes, submit a ticket to contact the technicians for help.
    1
    An exception occurred during Flink initialization, resulting in startup failure.
    This is generally caused by basic conflicts or overwriting of critical configuration files. You can search logs by Could not start cluster entrypoint and view relevant exceptions.
    If no cause can be identified, submit a ticket to contact the technicians for help.
    2
    A fatal error occurred during the startup of the Flink JobManager.
    Search logs by Fatal error occurred in the cluster entrypoint and view relevant exceptions.
    If no cause can be identified, submit a ticket to contact the technicians for help.
    239
    An uncaptured fatal error occurred in Flink execution threads.
    Search logs by produced an uncaught exception. Stopping the process and other keywords and view relevant exceptions.
    If no cause can be identified, submit a ticket to contact the technicians for help.
    
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support