LogListener Service Logs

Last updated: 2021-07-20 11:57:20

    Overview

    LogListener service logs are used to record the operation, collection, and monitoring activities of LogListener, and you can configure visualized graphs to display such log data.

    Default configuration

    Default Configuration Item Description
    Log Topic When you enable LogListener service logs, the logset cls_service_logging will be created for you automatically, and all log data generated by associated machine groups will be categorized and stored in corresponding log topics. The following three log topics are created by default:
  • loglistener_status: the heartbeat status logs for the corresponding LogListener.
  • loglistener_alarm: logs corresponding to LogListener's monitoring by collection metric/error type .
  • loglistener_business: logs corresponding to LogListener's collection operations, with each log corresponds to one request.
  • Region After the LogListener log services are enabled, logsets and log topics will be created under the machine groups in the same region of LogListener.
    Log Storage Duration The default storage duration is 7 days, and the value cannot be modified.
    Index Full-text index and key-value index are enabled for all collected log data by default. You can modify the index configuration. For details, please see Configuring Index.
    Dashboard Dashboard service_log_dashboard will be created in the same region of LogListener by default.
    Note:

    • LogListener service logs are only about collection and monitoring activities of LogListener, and do not support writing into other kinds of data.
    • Log data generated by this feature does not incur costs.
    • cls_service_logging is a unified logset for LogListener service logs.

    Operations

    • Viewing LogListener status
      When the LogListener service logs are enabled, you can view LogListener running status and collection statistics. You can view the number of active LogListeners, LogListener status distribution and other statistical metrics on the service_log_dashboard dashboard.

    • Log collection monitoring configuration
      You can configure the collection and monitoring service logs by metric/error type. For example:

      • MEM, CPU, collection speed, collection latency, or other metrics
      • The number of parsing errors of LogListener
    • File-level monitoring
      With the LogListener service logs are enabled, you can view the monitoring logs of files and directories. For example:

      • All collection statistics files of one IP
      • The amount of logs collected under a certain path on a certain IP, such as app1 application logs located in /var/log/app1/. You can get the statistics of logs collected under this path.
      • The collection statistics of a topic.

    Prerequisites

    Only LogListener v2.5.4 and above support collection and monitoring service logs by machine/machine group. You’re advised to upgrade LogListener to the latest version.

    Directions

    Enabling the service logs

    1. Log in to the CLS console.
    2. In the left sidebar, click Machine Group to go to the machine group management page.
    3. On the machine group list page, select the target machine group and click to enable the LogListener service logs.

    Disabling LogListener service logs

    1. Log in to the CLS console.
    2. In the left sidebar, click Machine Group to view the machine group list.
    3. Select the target machine group, and click to disable the LogListener service logs.
    Note:

    After this feature is disabled, the log data saved in the logset cls_service_logging will not be deleted automatically. You can manually delete the logset where the service logs are saved.

    Dashboard of Service Logs

    When the LogListener service logs are enabled, CLS will create a dashboard search_log_dashboard by the type of recorded logs to display LogListener’s collection and monitoring statistics.

    Collection statistics dashboard

    You can go to the Dashboard page of the CLS console, click the ID of the target dashboard to view LogListener collection statistics, including its status, parsing failure rate, sending success rate, and other metrics.

    Log Types

    LogListener status logs

    The parameters of the log topic loglistener_status are detailed as follows:

    ParameterDescription
    InstanceIdLogListener unique identifier
    IPMachine group IP
    LabelAn array of machine IDs
    VersionVersion number
    MemoryUsedMemory utilization of LogListener
    MemMaxMemory utilization threshold on this machine set by the Agent
    CpuUsageLogListener CPU utilization
    StatusLogListener running status
    TotalSendLogSizeSize of logs sent
    SendSuccessLogSizeSize of successfully sent logs
    SendFailureLogSizeSize of sending-failed logs
    SendTimeoutLogSizeSize of logs with sending timed out
    TotalParseLogCountTotal number of logs parsed
    ParseFailureLogCountNumber of parsing-failed logs
    TotalSendLogCountNumber of logs sent
    SendSuccessLogCountNumber of successful sent logs
    SendFailureLogCountNumber of sending-failed logs
    SendTimeoutLogCountNumber of logs with sending timed out
    TotalSendReqsTotal number of requests sent
    SendSuccessReqsNumber of successful sent requests
    SendFailureReqsThe number of sending-failed requests
    SendTimeoutReqsNumber of requests with sending timed out
    TotalFinishRspsTotal number of RSP files received
    TotalSuccessFromStartTotal number of successfully sent requests since LogListener was enabled
    AvgReqSizeAverage request packet size
    SendAvgCostAverage sending time
    AvailConnNumNumber of available connections
    QueueSizeThe size of queued requests

    LogListener alarm logs

    The parameters of the log topic loglistener_alarm are detailed as follows:

    Monitoring Metric Description
    InstanceId LogListener unique identifier
    Label An array of machine IDs
    IP Machine group IP
    Version LogListener version
    AlarmType.count Statistics of alarm types
    AlarmType.example Sample alarm type

    AlarmType:

    alarm type type ID Description
    UnknownError 0 Initializing the alarm type.
    UnknownError 1 Failed to parse.
    CredInvalid 2 Failed to verify.
    SendFailure 3 Failed to send.
    RunException 4 Abnormal LogListener running.
    MemLimited 5 Reached the memory utilization threshold.
    FileProcException 6 Exceptions occurred in file processing.
    FilePosGetError 7 Failed to get the file publishing info.
    HostIpException 8 Exceptions occurred in the server IP thread.
    StatException 9 Failed to get the process info.
    UpdateException 10 Exceptions occurred in the CLS modification feature.
    DoSendError 11 Failed to confirm sending.
    FileAddError 12 Failed to create the file.
    FileMetaError 13 Failed to create the metadata file.
    FileOpenError 14 Failed to open the file.
    FileOpenError 15 Failed to read the file.
    FileStatError 16 Failed to get the file status.
    getTimeError 17 Failed to get the time from the log content.
    HandleEventError 18 Exceptions occurred in processing the file.
    handleFileCreateError 19 Exceptions occurred in handleFileCreateEvent().
    LineParseError 20 Failed to parse the log directory.
    Lz4CompressError 21 Failed to compress.
    readEventException 22 Failed to read.
    ReadFileBugOn 23 A bug exists.
    ReadFileException 24 Exceptions occurred in the read file.
    ReadFileInodeChange 25 File node changed.
    ReadFileTruncate 26 The read file is truncated.
    WildCardPathException 27 Exceptions occurred in addWildcardPathInotify().

    LogListener collection logs

    The parameters of the log topic loglistener_business are detailed as follows:

    ParameterDescription
    InstanceIdLogListener unique identifier
    LabelAn array of machine IDs
    IPMachine group IP
    VersionLogListener version
    TopicIdThe target topic of the collected file
    FileNameFile path name
    FileNameActual file path
    FileInodeFile node
    FileSizeFile size
    LastReadTimeThe most recent read time of the file
    ParseFailLinesNumber of parsing-failed logs within a time window
    ParseFailSizeSize of parsing-failed logs within a time window
    ParseSuccessLinesNumber of logs successful parsed within a time window
    ParseSuccessSizeSize of logs successful parsed within a time window
    ReadOffsetOffset of file reading in bytes
    TruncateSizeSize of truncated log files within a time window
    ReadAvgDelayAverage time delay for reads within a time window
    TimeFormatFailuresLinesNumber of timestamp matching errors within a time window
    SendSuccessSizeSize of logs successful sent within a time window
    SendSuccessCountNumber of logs successful sent within a time window
    SendFailureSizeSize of sending-failed logs within a time window
    SendFailureCountNumber of sending-failed logs within a time window
    SendTimeoutSizeSize of logs with sending timed out within a time window
    SendTimeoutCountNumber of logs with sending timed out within a time window
    DroppedLogSizeSize of dropped logs within a time window
    DroppedLogCountNumber of dropped logs in a time window
    ProcessBlockWhether the current file has triggered collection blocking in a statistical period (collection blocking will be triggered if the sliding window of a file has not moved for 10 minutes)